Nati Shalom's Blog

The Disruption Cycle: A Dime a Dozen

 

Before the dust even settled around the whopping Dell/EMC deal, another bombshell was dropped with the Red Hat acquisition of Ansible over the weekend.  While I was still processing my thoughts on what I think the Dell deal means for the enterprise IT industry - I had to make an abrupt about-face and think about this poster-child of the community and bottom-up startup approach of Ansible means for the IT industry as a whole, and how this may be an even more substantial win. 

The EMC/Dell deal seems to be centered more on the merits of a financial transaction, i.e. grabbing billions of dollars from the existing customer base, refunding debts, latching onto VMware as the only publicly traded company and so forth.  The following diagram summarizes this quite nicely:

Dell Acquisition of EMC (and associated Federation assets) – blue text is new [Source: Wikibon, 2015]

What seems to be clear is that while the EMC/Dell deal is good for its shareholders its equally bad news for their customers as noted by the David Vellante report on Wikibon as both companies will have to squeeze their customer base to justify the ROI for the deal.

“Dell is promising within 18-24 months it will de-leverage (i.e. pay down its debt). How do companies manage cash cows? Answer – they leverage their lockin: 1) Longer product cycles; 2) Increase maintenance/support costs; 3) Cut salaries; 4) Narrow the R&D focus; 5) Charge a premium – We expect this to be the likely Dell playbook for VMAX, VNX, Data Domain, Compellent, etc”

It's hard not to notice that there's nothing in this deal that tells a clear story on how the new company will transform into the new cloud/web-scale business model.

So if anything I think that the Dell/EMC deal marks an end of an era more than a new beginning as Cade Metz puts this nicely on wired.com

“HP. Cisco. Dell. EMC. IBM. Oracle. Think of them as the walking dead. Oh, sure, they'll shuffle along for some time. They'll sell some stuff. They'll make some money. They'll command some headlines. They may even do some new things. But as tech giants, they're dead...When someone asked what we should call that IBM-HP-EMC-Dell-Cisco merger, his response was wonderfully descriptive. He suggested we call the company Fucked By The Cloud.”

It's also interesting to read the reaction to this post on twitter :100:

The case for Red Hat/Ansible is the exact antipode of the EMC/Dell deal. The financial factor didn't really play any real factor at all, what really mattered is the developer mindshare, adoption rate and popularity of Ansible in the DevOps community.

The Ansible stats tell the story - plain & simple - high adoption rate, low revenue, and I would add, in essence a high adoption rate at the expense of revenue.

• Founded in 2013

• 40 people

• $6M investment round

• High & rapid adoption

• Low sales - Estimated to be ~$1M

• Acquired at $150M

The ROI on this deal is a no brainer for Ansible.  The rationale for the Red Hat acquisition isn't so much on the direct revenue from Ansible as a product, but rather the potential monetization from its  impressive adoption.  This is interesting as it clearly put the benchmark for a successful startup these days back on eyeballs and popularity, and less on having a sustainable business model. But this seems to be the new game, and technology companies are all scrambling to get in line with this new model.

What's also interesting is that we’re starting to see new companies emerging to disrupt not just the dinosaur of the world, but also the early cloud companies such as Puppet and Chef that are now being disrupted...and there's a long list of such companies. Salesforce is a good example for one that is also being threatened by this rapid disruption.

We can see a clear vicious cycle that applies to all companies old and new. New generation companies rise on the wave of adoption by providing free and open source tooling, as they start to monetize and build a sustainable business model they start to lose popularity, and a call for a new disruptive tool is heard.

That explains the timing behind this acquisition, Ansible had reached a ceiling when it comes to adoption, and failed to build a sustainable business model out of their product. Containers/Docker have disrupted the entire configuration management market, and to a large degree render it obsolete (the adoption curve shows a clear declining trend on all the other configuration management tools).

With this in mind Ansible is clearly at its peak right now, so their best option was to be acquired by a bigger company like Red Hat. Red Hat is probably the only company with a sustainable business model around open source, so for them the direct revenue from Ansible isn't that interesting but more of the combined revenue that could be generated by bundling Ansible into a bigger DevOps tool set. I'm sure that the fact that Ansible’s founders are also formerly Red Hat played a not small factor in the overall decision.

Final words

What have we learned from all this?

• Cisco, EMC, Dell, HP, IBM are “walking dead”

• Adoption/popularity is the new king

• The only constant is change!

Disrupt yourself before your competitors do --> Embrace change

Steve Jobs’ main mantra was "disrupt yourself before your competitors do" and indeed what's clear in both deals is that change is inevitable, and it can and will happen to big companies just as it can happen to a smaller company. Therefore, the assurance that we used to think we had by relying on big companies isn't a safe bet anymore, similarly the risk of using new technologies from startups isn't as great as it used to be.

In this world those who would survive are only those who are set to adopt new technologies fast. There’s a direct correlation between the speed of adoption and the speed of innovation. The challenge is actually even bigger for companies who grow fast. Chef and Puppet are a good example in this regard. They created a new disruption, grew fairly fast, but stagnated for a long time, and failed to respond to the new disruption in their domain. 

This is where I see the biggest shift in the value chain specifically in the new data center. The value is less on how we build the tools, but more on how we put them together, and how we can adopt new tools or replace existing ones fast in way that will allow us to always be ready for the next disruption. This explains the rise in orchestration tools, as I pointed out at the last OpenStack Summit, and in my previous post orchestration tool roundup. that plays an important role as a "gluing" mechanism in the area of DevOps and automation.

Making Hybrid Cloud Simple and Real with vCloud Air, TOSCA and Cloudify

 

 

While hybrid cloud has been a mainstay discussion in the cloud world for quite some time now (note this post of mine on hybrid cloud from 2011, when CloudStack was still Cloud.com), the reality is that setting up hybrid cloud has proven a fairly complex process. Therefore we've seen only a few real implementations of true hybrid cloud in the wild.

In this post, I  wanted to describe the joint work that we've invested together with the VMware cloud team to change this reality and finally make the hybrid cloud story simple and possible.

vCloud Air makes hybrid cloud truly simple and real

While most public clouds started as independent public cloud services, and only later started to add private cloud connectivity; the exact opposite is true for vCloud Air, which was designed primarily as an native extension of VMware private cloud. In this context, users are able to use the same tools currently in use to manage their local VMware environment in order to manage their public cloud resources, as well. This makes the hybrid cloud story significantly simpler, as there is no longer the need to worry about the connectivity between the two sides of the data center, and at the same time we get the cost benefit of using resources on-demand on a pay per use basis.

The ability to make hybrid cloud so simple fit well with the Cloudify pure-play orchestration vision, and made the integration with the two platforms a perfect fit.

In this post I wanted to spend some time on what that means for VMware and OpenStack users.

Putting vCloud Air, TOSCA and Cloudify  together

 

vCloud Air users needed a simple way to deploy and manage their applications on vCloud Air using DevOps tools similar to Amazon Cloud Formation.

As the landscape is still being shaped, and there’s no clear winner yet, it is very important for these users to have tools that will allow them to keep their options open.

TOSCA seemed to be a great fit in such a context, as it provides a standard templating language that doesn’t tie users into a particular platform.

On top of this, TOSCA orchestration also aims to go beyond the installation part of the application and covers all the aspects of the application lifecycle through the addition of workflows and policies that can be used to handle continuous deployment, self-healing, and auto-scaling processes.

On a similar note, many Cloudify users comes from Enterprise and Telco markets. Such users are already heavily invested in VMware. These customers are actively looking for a more agile and cost effective way to run their data center.  The ability to bring capacity on demand seamlessly into private cloud makes the entire private/hybrid cloud story truly attainable.

By putting vCloud Air, TOSCA and Cloudify together we’re getting the cost benefit of using on demand resources for our private data center in a way that doesn’t tie us to a specific platform and at the same time provide a more complete application lifecycle and management solution that enables full automation of our deployment and DevOps processes.

Extending the solution to the rest of the VMware stack

VMware provides three kinds of platforms today, the vCenter/vSphere environment that serves many of the enterprises data centers today, and VMWare Integrated OpenStack, which provides API compatibility with OpenStack for vSphere/vCenter environments.

Cloudify comes with native TOSCA support through plugins that allows you to use TOSCA blueprints as a templating language across both vCloud Air, vSphere and OpenStack that provides a consistent way to manage applications across all three environments.  On top of this, it also makes it simple to build mixed resources of all three environments all with the same blueprint.

Having pure-play orchestration and management that is backed by an industry standard, TOSCA, enables the decoupling of the way to manage applications from the underlying infrastructure. While there is still the need to rely on specific APIs and features of the specific infrastructure, they are now at a point where the cost of the switch isn’t as significant compared with the alternative of binding themselves directly into the specific underlying infrastructure.

The benefit for OpenStack users

Many OpenStack users are Enterprises and Telco customers who are known to be heavy users of VMware platforms. This integration makes the ability to mix and match existing VMware environments together with OpenStack and vCloud Air, providing greater flexibility to decide which applications and workloads fit best in which environment while using common and consistent management for their applications across environments.

What should we expect next?

Through our partnership with VMware we’re planning to make this integration simpler for vCloud Air users by allowing them to use TOSCA directly through the vCloud Air public cloud service. In addition, the plan is to make the hybrid cloud story between vCloud Air, vSphere and OpenStack as seamless and simple as possible so we will be adding more examples and tighter integration for all three platforms.

How should I get started?

• TOSCA Orchestration for VMware - Hybrid Cloud Made Simple
• vSphere Plugin Docs
• VMware Hybrid Cloud with Cloudify

OpenStack by the Numbers

One thing that I like about working with an open source community like OpenStack is that you get direct access not just to the source code itself, but to its users and ecosystem as well.  This allows you to be less exposed to marketing spins, and actually analyze our actions based on real numbers.

The OpenStack Foundation conducts a very insightful survey twice a year which helps to measure how users are using the technology. This is an important and useful feedback loop that goes to the PTL of the various projects, and can help product companies to analyze their go to market plans and such. You can find the source

I found the original version a bit too verbose, so I took the liberty to put together an executive summary of the main points from the survey, mostly targeted at decision makers. I hope that you’ll find it useful, any feedback would be highly appreciated.

Main Reasons for Choosing OpenStack

The main reason cited for choosing OpenStack is Open Technology and Avoiding Lock-in.

Interestingly enough Cost Saving comes only 4th!

 

Size of Deployments

The large majority of the current deployments are under 100 nodes.

Around 30% of the deployments run relatively large scale deployments between 500-1000.

QA/DEV	Production

Production seems to follow a similar trend but lower in numbers to QA, which is reasonable as I believe that some of the QA deployments represent a transition stage to production.

 

User Type

The majority of the users seem to be service providers and cloud operators.

There is also a good number of ISVs and ecosystem partners, which is a strong indication that OpenStack is indeed an ecosystem.

There's a very strong presence of OpenStack in the Telco industry. Finance seem to be lagging behind the adoption curve with only three financial organizations declaring their use of OpenStack.

The majority of the deployments run on private or hosted private clouds. About 20% run on public or hybrid clouds. OpenStack also seems to attract mostly small to mid-size companies.

 

Private/Public Clouds	Size of Organizations

 

Deployment Tools

The survey includes two categories for deployment tools, one for those that are used for deploying the OpenStack infrastructure, and the other that are used for application configuration (covered in the application survey).

According to the two surveys users tend to use different sets of tools for the infrastructure deployment and for application configuration management. Puppet seems to be the most popular choice for OpenStack users in both categories. 

One of the interesting results is to see that Bare Metal and Docker are starting to appear in more deployments. This is fairly impressive given that the two are not yet officially supported by OpenStack.  VMware is around 10%-13% of KVM of Virtualization similar to XEN.

 

OpenStack Deployment Tools

 

Applications on top of OpenStack

As noted above, Puppet is popular for application configuration, as well as infrastructure deployment. Heat and Docker take the 2nd and 3rd place respectively,  putting Chef behind in the 4th place.

Given the popularity of Heat during the last summit I wouldn’t be surprised if Heat and Docker will surpass the use of Puppet in this category.

 

 

Popular Frameworks

Interestingly enough, JClouds seems to be gaining popularity amongst OpenStack users. This may be an indication that ~ ¼ of the users are interested in frameworks that are not tied only OpenStack. Not surprisingly, Python is far more popular than any other language, however Java seems to have a good and dominant presence more than other frameworks. One possible explanation is that the type of users that are attracted to OpenStack are more of the traditional enterprise organizations, and less of the new web-facing companies.

Operating Systems

Ubuntu and CentOS are the most popular choice. Windows adoption amongst OpenStack users is between 10-20%.

<

Application Survey	Production Survey

 

Compatibility / Portability

EC2 is a relatively popular choice which indicates that there is an increasing desire among the OpenStack community to maintain portability with EC2. This also correlates with the popularity of portability frameworks such as JClouds.

 

QA/DevTest	Production Survey

 

References

• OpenStack user survey presentation
• OpenStack user survey online session from OSS Atlanta 2014

Making Chef as Simple as Ansible

In his book Taste Test, Matt Jaynes compares a few popular configuration management tools such as Chef, Puppet, SaltStack and Ansible.

Jaynes points to some of the areas of complexity within Chef in comparison to Ansible. According to Jaynes, Chef’s areas of complexity can be summarized by the following:

• Ease of Installation: Chef is not easy to install

• Agent Installation: Chef doesn’t provide a remote agent installation 

• Remote Execution and Update: Chef lacks remote execution and push update

• Ordering: Controlling the execution order of tasks is not intuitive

Making Chef as Simple as Ansible using Cloudify

One of the things that has kept me busy lately is how to make the experience of using Cloudify and Chef a smooth and simple experience. We wanted to get to the point where users can run a Chef Server or Agent, install cookbooks and wrap it with Cloudify Application Service in just a few minutes without having to download or install any software.

We've launched a new online service that does just that – making Chef on the cloud simple.

The way this works is that once you click on the launch button a new server is created on the cloud (HP OpenStack cloud in this specific case). Next we install the Chef-Server using Cloudify on that machine. After 2-3 minutes once the server is up a new interactive shell window is opened. The interactive shell will allow you to interact with the service by invoking a set of custom commands that are listed just below the interactive shell screen.

If you want to view the progress and process of deployment you can click on the “Manage Your App” button. You can also click on the “Use Chef” to open the Chef web admin.

While we wanted to make the experience simple it was equally important that we wouldn’t take shortcuts in that process for the sake of a demo. To that end, we kept the demo identical to the way you would run things in production. That means that all that is needed to build a similar demo yourself is download Cloudify and follow the same steps in your own environment.

Learn more about how Cloudify makes the Chef experience simple

Here’s how Cloudify maps specifically to the challenges outlined in Matt’s book.

• Ease of Installation: Cloudify comes with built-in server provisioning support that works with a wide range of clouds out of the box, namely OpenStack, AWS, Azure, CloudStack, among others. Cloudify also comes with built-in integration with Chef. Cloudify allows you get your Chef Server installed and running on any of the supported clouds. 

• Agent Installation: Cloudify will provision your application machines, install a Chef agent and wire them up with the Chef server. Or, if you choose to run solo, it will allow you to install the agent with the appropriate cookbooks for that service.

• Ordering: Cloudify allows you to define the dependencies between services and will launch them based on that order.

• Remote Execution and Updates: Cloudify uses a custom-command abstraction that allows you to execute remote shell scripts on a group of instances that belong to a particular service or to a specific instance in a group. This feature is useful to run maintenance operations, such as snapshot in the case of a database or code push in a continuous deployment model.

 

Remote Execution with Cloudify

 

Beyond Deployment Automation

While improving your deployment experience with Chef is reason enough to use Cloudify, there are other benefits to gain as well. The real long-term benefit of using Cloudify is to manage post deployment through built-in management, logging and monitoring capabilities, as well as the built-in workflow for automating failover and auto-scaling processes.

Cloudify 3.0 Monitoring

 

Technical Overview of the Cloudify and Chef integration

To learn more on how Cloudify works and how you can build your own Cloudify/Chef service see the Cloudify Chef integration documentation.

 

What’s next with Cloudify 3.0

The upcoming release of Cloudify (currently in Beta) includes further enhancements - specifically the support of Yaml/TOSCA-based configuration. Using Python to reduce the agent and server footprint, allows custom workflow setup to allow users to add their own set of rules to automate upgrades, fail-over and scaling processes, enhanced monitoring and many other goodies, so stay tuned for further updates shortly..

Final Notes

If you're a Chef or Cloudify user, please share your experience  with me with this new service and, more importantly, let me know if there are areas where we can do even better. I would be most appreciative of your feedback.

Network Automation and Orchestration with Openstack and Cloudify

In my previous post I discussed the evolution of networking in the cloud - from commodity hardware to the cloud, and the need for the virtualization of network functions, and the different drivers for change.  In this post I will focus on orchestrating your networking through OpenStack’s Neutron, its challenges and potential.

Bringing the Network and Applications Together

So, as I mentioned in my previous post, when OpenStack introduced its networking API, known as Neutron, this provided a new way to set up networking services and configurations such as routers, subnets, floating IPs, security groups, and more through a simple API.  Until now we used to think of network configuration and application deployments as two separate things. This also goes back to the way most of our IT is structured.  We have different groups for networking operations and for applications’ operations and in most cases they have a fairly loose interface between one another.

As with DevOps once we open our infrastructure layers through APIs we can break those traditional boundaries between networking and apps. In a networking context, this means being able to create and configure our network as part of the application deployment.  To understand what this really means, let’s examine what the typical steps that we need to follow to set up networking for our application.

Typical Network Configuration:

1. Set up admin network for our application monitoring and control services
2. Set up application network to allow internet access to the relevant machines
3. Set up the security group per application tier (open only the ports and IPs relevant for that tier)
4. Set up floating IPs where needed
5. Set up a load balancer and configure it to route traffic to our web containers

Adding Continuous Availability

There are a set of additional network configuration steps that needs to be considered when we think of high availability.

Such steps often include:

1. Partitioning our services into separate availability zones and regions
2. Setting up connectivity between the availability zones and regions (which often run on separate networks)
3. Switching back and forth between each availability zone in case of a failure

Orchestrating and Automating  Network Configurations as a Part of Application Deployment

Today most of the steps required to set up our network are often done manually and externally to our applications. Now that we have API access, and our networking is basically software-driven, we can automate this entire process.  As many of those steps tie in more directly to our application services, we need to include them with our application deployment.

Network Orchestration Through a Simple Blueprint

With an orchestrator, users can define their own gateways, routers, and security groups through simple software configuration files, or blueprints, that are written in a language that the orchestrator (software operator) can understand.

With the latest Cloudify 2.7 release we added the ability for users to set up their network configuration as part of their application deployment blueprint. This allows users to set up security groups, separate networks, isolate the management network from the application’s network, define whether a certain piece of your application will run in the same availability zone or not, control if the storage will be close to the application geographically (i.e. the affinity rule).  Read about Network Automation to see how such a blueprint should look.

Final words

Bringing the application and networking together provides an easy way for users to capture the value of newly available network capabilities without having to become a network operator themselves. This significantly simplifies the complexities involved with exposing networking elements to the user by abstracting the networking piece of our deployment into an application blueprint.

In addition, the fact that we can setup our network configuration as part of our application deployment allows us to harden our security to match our application topology exactly . For example, we can choose to open only the necessary nodes to internet access, setup security groups dynamically to match the ports and IP addresses that are allowed to access our service and other such customizations. In this way, setting up an application and its network configuration is infinitely simplified for an application developer.

Eight Cloud and Big Data Predictions for 2014

 

Prediction has always been a fun exercise. It forces you to take a step back from your day to day activities, look at the market "crystal ball" and figure out what the future looks like. This year I found this exercise to be particularly difficult as the amount of new innovation and conflicting trends that are taking place at the same time can be very confusing.

After spending quite some time reading market analyses from different sources (see references at the end) and compiling them with all the things I've seen and heard throughout the year, I came to the following observation on how 2014 will look for Cloud and Big Data.  I'm happy to exchange thoughts on that regard and learn how you envision 2014 will look.

 

Public and Private Cloud in 2014

1. Amazon continues to distance itself while Google and Microsoft are catching up

Amazon continues to lead in public cloud and distance itself from the rest of the market. This year it reached 5 times the size of other cloud vendors combined, as noted in a recent Gartner report. Google and Microsoft are slowly closing the gap as the closest alternatives, but at a much slower pace than one would expect given the significant investment of both companies in this area.

2. 2014 will be the year of Enterprise Clouds

Traditional enterprise players, such as IBM, HP, Cisco and Red Hat, continue to fight for the remaining share of the market, mostly around enterprise adoption. Yet, enterprises have been slower to embrace and execute on private cloud strategy. Part of the reason for slow adoption is the gap between the solution provided by most of the regular contenders - who are still competing on selling an end to end story - and the reality that most enterprises are looking for Open and Hybrid cloud strategy, especially given that there is no clear winner.

3. OpenStack will be the most popular choice for Enterprise Clouds in 2014

OpenStack is now high on the radar for Enterprise Cloud, mostly threatening VMware's strong leadership in that domain. Most of the main contenders have embraced a strong OpenStack strategy, including Red Hat, Ubuntu, Suse, HP, IBM and Cisco, with Cisco taking a surprising leading position over IBM and HP, according to a recent Forrester survey.

VMware's response of embracing OpenStack is still questionable, as the transition to OpenStack is not only about technical integration, but also involves a big shift in the value chain. More enterprises are less keen on paying high cost for hypervisor licenses, which is to date the main revenue channel in the VMware pie.

4. Native OpenStack alternatives will disrupt many of the existing cloud solutions

The rapid adoption of OpenStack will also disrupt many of the existing cloud solutions that were built in a pre-OpenStack world. New solutions that take a more native-to-OpenStack approach will pop up and replace many of the current solutions. A good example of this is Project Solum, which aims to provide a native alternative to existing PaaS solutions, such as CloudFoundry and OpenShift as I outlined in this recent post. I expect that will see more of these disruptions expanding to other frameworks in 2014.

5. Orchestration & Automation will be the next big thing in 2014

Having said all that, the remaining challenge of enterprises is to break the IT bottleneck. This bottleneck is created by IT-centric decision-making processes, a.k.a "IaaS First Approach," in which IT is focused on building a private cloud infrastructure - a process that takes much longer than anticipated when compared with a more business/application-centric approach. One of the ways to overcome that challenge is to abstract the infrastructure and allow other departments within the organization to take a parallel path towards the cloud, while ensuring future compatibility with new development in the IT-led infrastructure.

DevOps has definitely been a key example for a business-led initiative that determines the speed of innovation and, thus, competitiveness of many organizations. The move to DevOps forces many organizations to go through both cultural and technology changes in making the biz and application more closely aligned, not just in goals, but in processes and tools as well.

Enterprises with legacy environments and customers take a two-step approach. They first tackle continuous delivery - automating the packages of their software deliverable and keeping tighter control over new production rollout. Once enterprises feel comfortable with their process and environment, they will automate the entire deployment process into production.

Configuration management, orchestration and workflow automation become key enablers in enterprise transition to cloud, and will gain much attention in 2014 and 2015. Again, Amazon has already recognized that need, introducing into the space a new offering called OpsWorks. It is only a matter of time until will see similar offerings embedded with other cloud providers in both public and private clouds.

As the focus moves to orchestration with greater focus on standardization of deployments and packages, DSL and API become equally important. Existing standards, such as TOSCA and CAMP, will undergo massive modifications to make them simpler to use and fit the cloud environment.

Big Data Predictions

There are many advancements happening within the Big Data/NoSQL domain that I'm not going to touch on. I will focus on two main areas that are close to my work.

6. 2014 will see a major increase of Big Data in the Cloud

Cloud infrastructure is closing the gap with the existing data center, as we see the emergence of support for Bare Metal, High CPU, High Memory and Flash-Disk. Many cloud providers are already offering Big Data as Service, such as Elastic Map Reduce and Redshift, and are continuously expanding their offering on that regard.

This advancement in cloud infrastructure removes almost all of the technical barriers for running I/O intensive workloads, such as Big Data analytics, on the cloud. I expect that in 2014, running Big Data analytics on the cloud will become the first choice for any new project, while the use of Big Data in non-cloud environments will be minimized to only niche use cases with extreme regulations and security constraints.

7. In-Memory Data backed by flash-disk will become a popular choice for Real-Time Big Data analytics
Flash disk pricing changes the economics behind the cost/performance ratio of disk-based solutions, making it possible to achieve the performance of an In-Memory-based solution at a price closer to a disk-based solution. So far, attempts have been made to use flash disks as a fast disk alternative to magnetic drives. This path inherits many of the limitations of a disk-based drive and, therefore, doesn't capture the full potential of flash disk which, like RAM, can provide highly parallel access to data.

At the same time, memory-based solutions like In-Memory Database or In-Memory Data Grid cover a small niche in the Big Data ecosystem, mostly due to the high cost/performance ratio.

I believe that the combination of flash drive, In-Memory data-grids and databases at the front-end change that dynamic and will make memory-based solutions backed by flash disk much more attractive to a bigger niche, specifically as it relates to real-time analytics of Big Data.

In some cases, the combination of memory-based solutions is also integrated with existing Big Data frameworks and, thus, provides seamless performance acceleration. A good example is GigaSpaces' integration with Storm and GridGain's integration with Hadoop.

8. Real-time analytics turns mainstream

The most interesting indication that real-time analytics is becoming mainstream is Amazon's support for real-time analytics, with many of the existing analytics solutions providing real-time analytics capabilities as built-in parts of their reports. Another good example of that is Google Analytics Real Time View.

New disruptive forces in Cloud worth watching in 2014
Disruptive technologies change the market landscape in ways that are difficult to anticipate by their very nature. Rather than predicting their impact, I felt it would be simpler to list them.

• Networking - The networking segment of IT is experiencing a major disruption as of late with the move to Software Defined Network, OpenStack Neutron project and Network Function Virtualization. These developments will change networking from not only a technology perspective, but they are also driving a completely different business model which will be based on utilization or a subscription-based model.

• Linux Container - Linux containers are gaining interest both as light-weight software packaging as well as VMS's. The most commonly used use case for Linux containers has as an underlying container for PaaS. With the introduction of new projects, such as Docker - which makes the Linux container easy to use, we will see wider and more pervasive use of containers as high performance virtualization, as packaging tools in continuous deployment scenarios, etc.

• Bare Metal Cloud - Bare metal cloud has been a small niche in the cloud space, mostly due to the fact that it is usually offered in a static configuration setup. Bare metal clouds provide the same degree of elasticity to bare metal devices. With OpenStack, for example, you can spawn a new bare metal device just as you would provision any other VM. The development would reduce one of the last remaining barriers for bringing mission critical applications to the cloud.

• OpenStack as an Innovation Accelerator - Many items on the list of disruptive technologies that I listed above are not that new, and to a certain degree, have existed for years, like in the case of Linux Container. Often, a disruptive force needs to gain certain critical mass before it can break into massive adoption. OpenStack creates an ecosystem that provides a platform for many users to integrate new technologies in a way that could be consumed by end users immediately and that plays a major role in the acceleration of adoption of many of those disruptive technologies.

Where do we go from here?

There are many individual technologies and trends that have a disruptive force behind them. Having said that, I think that it is the intersection between those technologies that holds the most promising potential. Here are few examples:

• Big Data and the Cloud Application Orchestration - Big Data analytics for operational information could serve as the new "brain" behind a new class of orchestration engine that would combine artificial intelligence decision-making based on trends and historical analysis and handle complex failure and scaling scenarios automatically.

• Putting Network and Applications together - Putting network and applications together holds a lot of promise in the way we scale applications across regions and multiple sites, as well as how we control application SLA's in a shared environment. For example, giving priority to customer-facing services versus batch analytics or optimizing the network routing based on the locality of the data, etc.

The Breakup: Why OpenStack Needs a Native Alternative to CloudFoundry and OpenShift

 

Alex Freedland, Co-Founder and Chairman at Mirantis, wrote a controversial post in which he basically marks the breakup between some existing PaaS frameworks and OpenStack - CloudFoundry specifically.

Personally, I think that Freedland highlights a growing sentiment in the OpenStack community. The fact that OpenStack is open source should change the way we think of the PaaS and IaaS layers, as I noted in one of my earlier posts, The Blurring Line Between the IaaS and PaaS Layers.

In the case of CloudFoundry and OpenShift specifically, PaaS was built with the assumption that the IaaS layer provides basic compute, storage and network services and all the rest should be handled at the layers above. The reality, though, is that the IaaS layer is continuously moving up the stack in a very similar way to which Amazon has grown its stack.

This shift up the stack changes many of the built-in design assumptions of many existing PaaS products, CloudFoundry and OpenShift specifically. For example, the way we handle user-security, metering and scheduling is often in a completely parallel stack that exists in both the IaaS and PaaS layers. This leads to a lot of unnecessary redundancy and therefore complexity and inconsistency. On top of that, the middleware stack (Ruby vs Python) also leads to another layer of redundancy and complexity, especially for organizations looking to deploy the end-to-end stack. For example, why would we need two different models to handle database availability? Why do we need different agent systems to monitor our instances? If we already have a model to detect and monitor device failure, why do we need to add another failure detection mechanism on top to detect the same failure? In many of these cases, we already built solutions at the IaaS layer and we only need to make the right use of them.

Now there is nothing wrong with CloudFoundry or OpenShift; they do exactly what they were designed to do. That being said, on OpenStack we can do things differently than we can on other closed IaaSs, creating a great opportunity to not just simplify things, but do do things that are much harder otherwise. Let me explain.

The Need for a Native Open PaaS

A native open PaaS is a PaaS that comes with a much thinner stack and relies more heavily on a common set of services provided by the IaaS layer for user management, metering, etc.

Its main focus is to provide a higher level abstraction that would simplify the deployment and management of applications on OpenStack.

Another area that a native open PaaS differs from its existing predecessors is the native DevOps support. Many of the existing PaaS platforms were built with the assumption that developers only worry about their code. However, in real life, continuous delivery or deployment can expose changes at all layers, including the configuration of the application and PaaS stack, and should therefore be consistent with the way updates and configuration management takes place in the IaaS layer.

A native open PaaS should leverage the same DevOps tools used by the IaaS layers to push updates to the application layer.

The Missing Piece in the Puzzle - Open DevOps Automation

Earlier this year, Amazon launched a new piece in their stack - OpsWorks, which basically filled a gap between their existing IaaS and PaaS layers. They refer to it as DevOps automation.

AWS Stack

 

AWS OpsWorks provides an orchestration and workflow service that currently runs on top of Chef and provides users with the ability to build their own custom stacks on top of AWS, as I noted in my earlier post DevOps, PaaS and Everything in Between.

The OpenStack Heat project fills the AWS CloudFormation box in the above image, providing a basic templating model for orchestrating and automating the deployment of software stacks on OpenStack.

I believe that OpenStack also needs  to fill in the OpsWorks box as higher level abstraction for application deployment. Given the open nature of OpenStack and the private cloud focus I think that this piece should be more open and support not just Chef, but also other configuration management frameworks such as Puppet, Ansible and Salt through Heat. There should also be a much stronger emphasis on the application orchestration piece to enable automation of complex application related processes, such as continuous delivery, deployment automation, recovery and scaling and for that we need a more advanced rule engine to support those advanced flows.

If done right, tools like OpsWorks in conjunction with Heat can serve as a great foundation for PaaS. At the same time, I can see many cases in which users would simply use something like OpsWorks directly without needing another layer of abstraction on top of it, hence why it makes sense to keep those boxes also as independent services.

Consolidating the Efforts

There are currently many projects and initiatives in the OpenStack community, each trying to address different aspects of the application deployment and management complexity. Let me name a few:

Adrian Otto, Principal Architect at Rackspace, announced Project Solum support from Rackspace, eBay, Red Hat, Ubuntu / Canonical, dotCloud as well as GigaSpaces.

Some of the guys from my team at GigaSpaces are also going to give a talk during the OpenStack Hong Kong event about delivering the equivalent of Amazon OpsWorks on OpenStack - See the details here.

There are many other related projects and initiatives that I haven’t listed here and many others that are probably being worked on as we speak - it is beyond the scope of this post to list them all.

Looking Into the Future

I think that this is a great and an exciting opportunity for the OpenStack community to deliver on its promise and really push the open source cloud forward. It can do so by breaking the traditional barriers between IaaS and PaaS, and go above and beyond what we've seen normally in closed source clouds. There are many areas in which such an approach can lead to an even greater degree of innovation. For example, think about bringing the network and the application together, making multi-site deployment a breeze.

I look forward to meeting everyone on the Solum team next week during OpenStack Hong Kong and to being a part of this next wave of the revolution.

References

• Project Solum announcement

• Deployment and Management of Your Application on OpenStack using an All Encompassing DSL based on TOSCA (OpenStack session in HK)

• Mirantis To PaaS Vendors: Stop Fighting OpenStack Gravity

• Rackspace jumps on bandwagon with open source PaaS project

• The blurring Line between the IaaS and PaaS layers

• DevOps, PaaS and Everything in Between

 

An Application-Centric Approach to DevOps

 

Matt Jaynes, Founder of DevOps University, wrote a great post on Hacker News, What is Continuous Deployment, in which he points out one of challenges that many people gloss over when they embark on their DevOps journey. In a nutshell, Matt was saying that, "You should get your house in order before aspiring to do automation or CD." In other words, if your testing is not fully automated or is taking too long to process, then having continuous deployment may not be right for you.

The thing that gets me so interested in this entire DevOps movement is that it forces you to think holistically about software development, business cycles and organizational culture. Some refer to this transition as the industrialization of IT. Indeed, there are many parallels between the shift from making cars by hand into a fully automated manufacturing pipeline and the DevOps movement.

With regards to Jaynes’ post, even if you can build a car faster, lack of proper QA may lead to shipping defective parts, and you will still be shipping slowly. In both options, your fast pipeline would end up producing diminishing return, as you may end up paying more for the warehouse or for handling "recall" or fixing defects, let alone the potential loss of reputation or even the entire business.

In short, you need to get your house in order before launching continuous deployment right into production.

Another area of complexity in building continuous delivery and deployment is that it often involves fairly complex processes for handling updates. I noticed that in many cases, people need to write a lot of custom code to implement these processes. That custom code can be fairly hard to maintain and track.

In this post, I would like to share a different approach to handling continuous delivery based on the experience of a Cloudify user - PaddyPower.

Paddy Power are a betting and entertainment company headquartered in Dublin Ireland.  Innovation has been the key to their success and from day one they sought to differentiate themselves from the chasing pack.  Like many organisations, they faced the need to speed up their development and deployment cycle. They went through a complete cycle of moving into agile development as well as transforming their infrastructure into a cloud-based infrastructure based on CloudStack.

They use Cloudify and Chef to run their DevOps automation and continuous delivery.

The way they approached DevOps in general, and continuous delivery in particular, is quite interesting because they took an application-centric approach to DevOps, as opposed to an infrastructure-centric approach as used in some other cases. Let me explain...

What is an Application-Centric Approach to DevOps?

In many DevOps discussions we often tend to jump into low level discussions about tools, and fairly quickly we start to lose sight of what we are actually trying to achieve.

Quite naturally, many start with one of the first steps in automation, which is configuration management - i.e. our ability to consistently configure, setup and update machines with the right stack. In this model, the world looks like a bunch of machines at the core onto which we then tie our software parts. I would refer to that approach as an infrastructure-centric approach.

An application-centric approach is looking at the same problem, but from a different perspective in which we start with a consistent definition of our application stack and blue-print (Load-Balancer, Web Container, Database, etc) and SLAs (failure detection, scaling rules, upgrade processes, etc). Configuration management comes as a natural complementary step for handling the installation and setup of our environment.

In this model, there is a greater emphasis on the orchestration of our application and its associated processes. A good example of such a process is continuous delivery. The execution of a continuous delivery process involves many steps that go far beyond a one-time execution of a given script. We often need to determine in which environment the script is going to be executed, which pieces of our stack are going to be included in each particular deployment, which configuration change is going to be included as well as code updates, and if there will be a database update- then which part will be affected by the change. Another example is continuous availability, automating the recovery process of our application in the event of failure. Such processes can become quite complex as we start to handle multiple availability zones, regions, as well as multiple environments for dev, test and prod.

With an infrastructure-centric approach, we often need to implement those processes into the code or scripts. Quite often, those scripts become very hard to maintain.

With an application-centric approach, we use a set of rules and events to handle the automation of those processes in a more generic fashion. With this approach, many of the policies and processes are handled more implicitly. For example, we can have a policy that if we fail during an update, the process will roll-back, or we can have a policy for creating an exact clone of our system for QAT, and only when it passes the full test may we start rolling it out, gradually moving traffic into those new processes.

Another difference between an application vs infrastructure-centric approach is in the level of abstraction that we expose to our developers. With an infrastructure-centric approach, each developer is often exposed to every part of our system. A continuous delivery model, where developers continuously interact with the system exposing the low level details of the infrastructure to every developer, can lead to complexity that is often unnecessary when all that is needed is to push code to an existing deployment. It can also lead to high vulnerability as a simple mistake can very easily bring an entire system down.

I thought that the following slide by John Turner, Software Development Manager at PaddyPower, does a good job of explaining how an application-driven approach through PaaS abstraction helps to address this challenge:

In summary, PaaS fits nicely into PaddyPower’s DevOps and continuous delivery strategy in the following ways:

• PaaS strives to abstract provisioning and deployment, thus reducing complexity.

• PaaS constraints applications to use features provided by the platform.

• PaaS is development focused.

In the case of Cloudify, the integration with Chef provides a great foundation for more open PaaS, one that fits nicely into DevOps environment. Chef cookbooks and recipes can be incorporated as part of the Cloudify definition of the application stack. It also fits well with the configuration and operational management of the platform.

Join me in our upcoming podcast with Ben Kepes of Clouderati and Diversity.Net, John Turner of PaddyPower and Andy Hawkins from Opscode as we discuss PaddyPower’s journey to app-centric DevOps. Watch it live via Google+ Hangout on Air | October 30th, 6:00 PM GMT, 10:00 AM PST.

What are the Specific Features in Cloudify that Makes it DevOps Oriented?

I will use Matt Jaynes’ definition from his book Taste Test:

• Directive & Directives Script -  Cloudify currently uses Groovy-based recipes with extended DSL as its directives. The Cloudify recipe allows you to combine Chef or Puppet directives as part of the Cloudify recipie. Our future release will include support for Yaml/TOSCA directives and may also include support for Ansible.

• Master Node & Children Nodes - Cloudify automatically bootstraps Master and Child nodes. It has a cloud plug-in that will allow you to automate this entire process and avoid any manual host/IP configuration. The Cloudify integration with Chef takes care of provisioning of Chef Master and Child nodes as well.

• Remote Execution - Cloudify uses custom-commands as a way to route calls to a group of individual sets of service instances. The integration with Chef includes built-in custom commands for updating Chef Coookbooks as well as for executing Chef-Knife and other Chef commands as described here.

In addition, Cloudify provides built-in services to query the state of the cluster through the ServiceContext API and share states across services in the cluster through the Attribute Store API.

Executing a Typical DevOps Task with Cloudify

1. Cloning a service or a complete stack - One of the features of Cloudify is the abstraction of its recipe (blue print) from the underlying infrastructure through a Cloud Driver. A recipe can point to a type of compute or storage node. The actual binding to the physical compute or storage resources takes place at runtime during the deployment process, based on the target environment. That means that you can easily clone a complete or only part of a given application stack across DEV, QA, PROD in a consistent way that will ensure that you use the exact setup between environments. The CloudDriver abstraction allows you to also adapt the size of the resource in each stage without touching the actual recipe - so for example a SMALL-LINUX machine in QA may point to a machine with smaller size and SLA than a SMALL_LINUX in production.

2. Updating a service with a new version - Custom-commands allow you to push updates to a particular instance or group of instances. Quite often, custom command calls will be triggered through a build system, such as Jenkins. Each command has access to the state of the cluster through the Attribute Store and ServiceContext and can, therefore, coordinate the update processes with the state of other services.

3. Orchestration flow - You can use the Cloudify Rest-API or Command Line to write custom flows. Certain flows, such as fail-over and auto-scaling, are built-in and in those cases you will only need to provide custom code to trigger for the flow.

Looking Into the Future

One of the things that we are working on right now is enhancing our orchestration capabilities, allowing users to choose more built-in flows or define their own custom flows that will be fully managed by the Cloudify orchestration engine.

We hope you join us for our live podcast on October 30th at 6:00 PM GMT / 10:00 AM PST on Google+ Hangout on Air.

References

• DevOps PaaS and everything in between.

• Its the apps stupid

• Application Centric Devops

• GigaSpaces Cloudify & VMware Cloud Foundry the new PaaS Jailbreaker

• Putting DevOps and PaaS together with Cloudify

 

To OpenStack or Not to OpenStack? Moving Enterprise Applications onto OpenStack Today

Peter Chadwick Senior Product Manager - Cloud Infrastructure at SUSE wrote an interesting article arguing  Why OpenStack Will Rule the Enterprise. Chadwick refers to three main points in his arguments:

• Variety - OpenStack is the only cloud platform that supports every major hypervisor.
• Familiarity - OpenStack follows an open source model similar to Linux. Most enterprises are already comfortable with that model and in that context OpenStack would be a natural evolution.
• Governance - The OpenStack Foundation was created to ensure that OpenStack evolves to meet a wide range of requirements that are not controlled by a single vendor.

Current Challenges for Enterprise Adoption of OpenStack

While there is a growing acceptance that this is the general direction, others seem to point out some of the gaps that still exist to enable enterprise adoption of OpenStack. Nancy Gohring mentions in an article in ITWorld Immaturity holding back OpenStack deployments  that the main challenges are the  lack of maturity and the fact that OpenStack is used mostly for testing and development and is not ready for true multi-tenant production deployments. She also points out the fact that many of the OpenStack distributions today are offered by relatively small companies, creating another barrier for adoption.  Bernard Golden more specifically highlights the upgrading and deploying of OpenStack as the main challenges for enterprise adoption in his article How OpenStack Should Prepare Itself for the Enterprise.

A Possible Strategy for Adopting OpenStack in an Enterprise World Today

All the cards indicate that despite some of the maturity and growing pains of OpenStack as a framework and community, it is heading in the right direction. But as we often say in those cases, "there is no fast forward for maturity." We need to expect that over the course of the coming years the reality is going to be fairly dynamic with lots of new developments coming out with every release and with many providers and players promoting their different OpenStack offerings. To compete with one another, these providers will also try to add their own value-added features as differentiators. So our proposed strategy has the following foundations:

• Design to cope with a continuously changing and evolving environment: We need to design our OpenStack environment in such away that we could adopt new versions of OpenStack  and new ecosystem frameworks easily.
• Design for portability across different OpenStack providers and versions: During a visit to one of our investment banking customers, he used an interesting analogy to describe his strategy for implementing an OpenStack-based private cloud on his environment:

“In a traditional data-center world we used to work at a hardware device and hypervisor level. We used different resources from Dell, HP etc. We’re thinking of doing the same thing in the cloud world, only that rather than working at the device and hypervisor level, we can work at the IaaS level with the hardware, storage and network pre-integrated. OpenStack allows us to standardize those providers and have a mix of them installed in our data-center."

In order to keep this level of flexibility, you need to ensure portability between the different OpenStack providers. This is especially true in a world where each provider tries to sell you the entire world wrapped up into one with OpenStack on top. If you design your data center to work with various providers in place, you're more likely to remain portable and avoid a complete lock-in.

• Use a high availability approach to enable simple upgrades without downtime: The simplest approach to upgrade any given system is to rip and replace it with a new one. Having said that, in most enterprises, taking a system down for the purpose of upgrades or maintenance is often not acceptable, for obvious reasons. If we design our system for continuous high availability, we could take down each of the cloud zones one at a time and still keep our application running. In this way, we could have a simpler upgrade approach; rather than trying to upgrade individual units of our infrastructure, we can can rip and replace an entire unit one at a time to continuously upgrade our system.
• Take a baby steps approach: As with any complex project, it would be more practical to implement OpenStack in small bits rather than taking a *Big Bang* approach. Here are few examples of what those steps could look like:
  • Start with automation first and future-proof your application to run on OpenStack at a later stage.
  • Run OpenStack for your development and testing environment.
  • Use OpenStack as your DR environment and gradually move your primary site to OpenStack as well.

Implementing the Strategy by Abstracting the Application from the Infrastructure

The points outlined above may seem obvious, but it is not yet clear how they can be implemented.

One of the ways in which we could implement portability, future-proof our application and allow for smooth upgrades is through the use of devops automation frameworks, such as Chef, Puppet or Cloudify, which itself integrates natively with Chef and  Puppet.

Such frameworks allow us to create an abstraction layer between our OpenStack infrastructure and our application to easily migrate our application across different OpenStack providers, versions etc.

Using Cloudify to Abstract your Deployment Between Different Private and Public OpenStack Providers

One of the challenges with many of the abstraction frameworks is the fact that they often rely on a least common denominator API approach. An API abstraction approach limits the level of unique features and capabilities that can be provided by each specific vendor.

In the case of Cloudify, we use a plug-in approach that is referred to as the Cloud Driver, which provides access to the IaaS layer, while a recipe model separately defines the application plan or blue-print that needs to be deployed.

With this approach, there is a basic contract/interface that each driver must implement to allocate compute, storage or network resources. Each driver can be given a set of specific properties and arguments with which the user can choose a specific network setup or compute API of a specific driver.  This approach provides a higher degree of flexibility than an API-based abstraction and we can, therefore, apply it to address a variety of scenarios as outlined below:

• Migrating between different OpenStack providers: The application deployment plan (recipe) is kept abstracted from this information. This way we can deploy our application on different providers using the same frameworks, but at the same time our application can still run better on a specific provider and utilize its specific set of features. 
• Migrating from an existing (non-OpenStack) data center to OpenStack: This abstraction allows us to work on top of our existing data center which may not yet support OpenStack. We can use this model to manage our application in the existing data center by pointing to a BYON (Bring-Your-Own-Node) CloudDrive, which gets a list or range of IP addresses as an input and manages the workload within that predefined pool.
• Migrating between different OpenStack versions: We can also use this model to migrate workloads from one version of OpenStack to the other. In this case, each provider's availability zone will represent a a specific OpenStack version. We can upgrade our underlying OpenStack zone just by removing it from the system. Cloudify in turn will automatically provision the application that was running on this provider in another zone which will be set to run the new version of our application.

Making Cloudify Native to OpenStack

One of the benefits of OpenStack as opposed to other cloud infrastructures is the fact that it is open and provides easy access to its underlying stack. We can, therefore, build Cloudify in such a way that utilizes some of the underlying OpenStack infrastructure, such as Heat for provisioning and Keystone for authentication. 

We can also standardize the existing component of Cloudify, such as the recipe, by adopting TOSCA and Heat notions and templates.

This way we can make the migration to OpenStack even easier. Not only are we enabling our application to run on OpenStack, we also do it in a way that will better utilize the underlying OpenStack infrastructure.

DevOps, PaaS and Everything in Between

This month Amazon announced AWS OpsWorks, which is the result of their acquisition of Peritor and the embedding of its Scalarium platform within AWS. OpsWorks is a new framework for simplifying the deployment of applications on the cloud.

Given that Amazon already has a few frameworks automating application deployment, such as CloudFormation and Elastic Beanstalk, I find the addition of OpsWorks to the mix to be an important recognition of the void that exists between the existing PaaS and Automation frameworks.

 

In this post I'll try to zoom in to that void between PaaS and DevOps.

I'll start with a post that I wrote a few months back - PaaS Jailbreaker, in which I pointed out the two models for delivering applications in the cloud, from PaaS to DevOps. In this post I suggested that rather than thinking of the two models as separate entities, it would make sense to combine the two.

James Urquhart wrote an excellent article on GigaOM - Is your PaaS composable or contextual? In his article, Urquhart refers to the differences between the two approaches to PaaS as composable vs contextual. A good example for a contextual approach is the plugin-based approach implemented by Heroku or Force.com. Plugins provide a flexible way to extend the platform, but the extension needs to fit into a certain model. For example, in the case of Heroku you don't control the actual VM or underlying cloud infrastructure, nor the network that is provided for that infrastructure. A composable approach, on the other hand, is focused on the composition of the different application components and, as such, doesn't need to assume anything about the underlying infrastructure that runs the components of our application. Amazon OpsWorks fits into that second category.

Urquhart specifically pointed out Dietzler’s Law for Access as a way to illustrate the limitation of the contextual approach.  While the contextual approach (customization via plugins) provides some degree of flexibility, it is the 10% of the things that you can't control that matters most, as they define whether you'll be able to be deliver your application the way you want to:

Every Access project will eventually fail because, while 80% of what the user wants is fast and easy to create, and the next 10% is possible with difficulty, ultimately the last 10% is impossible because you can’t get far enough underneath the built-in abstractions, and users always want 100% of what they want.

Interestingly enough, Amazon launched OpsWorks a week later.

The introduction of OpsWorks was quite interesting as it came two years after Amazon launched its Heroku/GAE equivalent - Elastic Beanstalk. I found the announcement by Amazon quite interesting. The fact that they had a PaaS offering already and still found it necessary to launch a new platform for delivering applications is:

1. Acknowledgement that the current PaaS offering isn't a one-size-fits-all solution, and despite the various extensions that Amazon delivered to support other languages within the platform, there are still many applications, and consequently organizations, that do not fit into the Elastic Beanstalk model.
2. Recognition of the fact that within the realm of the application delivery, there is a need for a more open framework for delivering applications, and that realm needs to take a more "composable" approach. In the case of OpsWorks, the composable approach happens to rely on Chef and the OpsWorks orchestration layer on top of it.

Werner Vogels uses the control vs convenience tradeoff as a way to position the various AWS offerings, as he pointed out in his post: Expanding the Cloud - Introducing AWS OpsWorks, a Powerful Application Management Solution:

With the launch of AWS OpsWorks customers now have a very powerful solution that allows them to manage their application easily without giving up control. 

In other words, OpsWorks is aimed at filling a void between the Beanstalk model and the DIY (do it yourself) model.

Interestingly enough, I used to describe the various solutions in the cloud stack in very similar terms to the way Vogels does, with the slight difference that I used the term Control vs Productivity and looked at a broader scope between IaaS and PaaS.

OpsWorks vs Other Kinds of PaaS

In a discussion I had on that subject last week with Ran Tavory and Ori Lahav as part of the Reversim podcast, we seemed to agree that currently, there is space for both models of PaaS - the OpsWorks model (composable) and the Beanstalk model (contextual). Having said that, when I framed the question "which of the frameworks/approaches will run 80% of the cloud workload in the future?" we all seemed to agree that the answer would more likely be OpsWorks. The last point led me to believe that while there is space for the two models right now, this may be only a transition stage, while over time we’ll see that first generation PaaS and the likes of Beanstalk will be replaced with second generation PaaS offerings, as in the case of OpsWorks.

OpsWorks and the blurring line between PaaS and IaaS

The fact that OpsWorks runs on Chef, as do many of the IaaS services in many cloud infrastructures, also raises the question whether it is right to draw a line between our IaaS services and applications, as I pointed out here: The blurring line between PaaS and IaaS.

OpsWorks and Cloudify

OpsWorks and Cloudify follow the same foundational concepts from an architecture perspective - both frameworks integrate with DevOps tools like Chef and add an orchestration layer on top of it, making it easier to compose various parts of the application stack. In many ways you can think of Cloudify as a framework for building your own OpsWorks.

PaaS, DevOps and Everything in Between

Vogels seems to avoid labeling either Beanstalk or OpsWorks as belonging to the PaaS category. Urquhart, on the other hand, looked at the two as two kinds of PaaS solutions - Contextual (Beanstalk) vs Composable (OpsWorks).

One thing that is clear to me is that the current PaaS category is too narrow to fit all the different kinds of PaaS.

Looking at the definition of PaaS in a broader context, I would frame it as a "Platform for running and managing applications on the Cloud." In that context, the way we choose to run those apps becomes an implementation detail. Having that context in mind, I tend to look at both Beanstalk and OpsWorks as two kinds of PaaS offerings. What we’re missing at this point is a simple way to label those different categories.

One approach would be to use Urquhart’s definition (“Contextual vs Composable) which categorizes the different PaaS solutions based on the approach through which they choose to deliver the apps. We can also look at CloudFoundry vs GAE as open vs closed. Another possible category is PaaS frameworks (i.e. frameworks that provide middleware to build your own PaaS), vs PaaS as a self-service solution.

RightScale, on the other hand, uses the term Cloud Application Management to define a similar category and looks at PaaS as a private use-case for using the Cloud Application Management framework. We also had internal debate on that subject at GigaSpaces and couldn't come to a final agreement on where to draw the line between the different categories.

What do you think? Is OpsWorks a new kind of (Composable) PaaS or Cloud Application Management Platform?

Does it really matter?

References:

• GigaSpaces Cloudify & VMware Cloud Foundry the new PaaS Jailbreaker
• Is your PaaS composable or contextual? (Hint: the answer matters)
• Mapping the Cloud/PaaS Stack
• The blurring line between PaaS and IaaS
• Expanding the Cloud - Introducing AWS OpsWorks, a Powerful Application Management Solution
• Amazon adds OpsWorks application life cycle management to AWS cloud