Nati Shalom's Blog

Do I Need PaaS if I Use Docker?

In my recent post on OpenStack  “Do I Need OpenStack if I Use Docker?” I covered the confusion that has emerged around one area of disruption that Docker has created. In this post, I’d like to segue into another similar topic that has emerged around the same concept where I sense similar disruption and confusion: the use of PaaS in its traditional form vs using a straight Docker approach.

Background

PaaS in its simplest form is a platform that enables developers to develop and deploy web applications into production fast by abstracting many details of the underlying infrastructure, whereas Docker provides a generic container for packaging software stacks in a portable way.

A Traditional PaaS Stack

Many of the traditional PaaS implementations use Docker or LXC as the underlying container to enable multi-tenant applications on a shared infrastructure. Usually PaaS relies on web containers that are specific to a certain language (Tomcat, Jetty/Java, modephp/PHP, Rails/Ruby, Django/Python,NodeJS/JavaScript…..) to host the user’s code, and Docker or LXC containers as lightweight VMs that hold these web containers, and provides the required isolation level between the various web containers. This architecture lends itself nicely into simple web applications which are often built out of a web front-end, and a database as a backend persistent model. As far as I know, Docker actually initially emerged from one of the PaaS’ provided by DotCloud, and evolved into its current incarnation.

The introduction of Docker as a simple abstraction to containers, has turned Docker into a portable software container that isn’t tied to a specific language or framework. It can not only be used as a web container, but also as a more general purpose software container for all sorts of applications such as microservices, big data analytics, stream processing, as well as legacy applications.

Some PaaS implementations, such as CloudFoundry, have already started to offer support for running Docker containers as one of the software components that are supported by the platform.

Having said that, PaaS provides a fairly opinionated model for the way we should model and package our applications. Often times, our packaging is too limiting as we start to grow to more scalable models, such as microservices.

This causes many users today to question whether there is enough added value in PaaS in order to justify these limitations, when users can simply run their software directly on Docker.

Using Docker as an Alternative to PaaS

Using Docker with a combination of orchestrations engines such as Kubernetes (in the case of GCE), Heat in the case of OpenStack, or Cloudify (for multiple clouds such as OpenStack, VMware, AWS, as well as bare metal)

can provide a relatively simple way to package, deploy and manage applications in an automated fashion.

Unlike the traditional PaaS stack - it doesn’t force a specific opinionated architecture for deploying apps although it does lend itself very nicely into both web applications and microservices architecture.

My Take

The main promise of PaaS is in the speed of development and simplicity. Traditional PaaS implementations achieve that goal through extreme abstraction of the underlying infrastructure, as well as by promoting an opinionated architecture, which provides a fairly optimized and consistent deployment experience for applications that fit into that model. The combination of the two ultimately does help achieve this promise, but for a fairly limited set of applications.

Using Docker coupled with orchestration as an alternative provides a more general purpose platform for achieving a similar goal. Unlike the traditional PaaS approach, this use of Docker with an orchestration engine does expose the user to more knobs and infrastructure details, which makes the user experience for developing and deploying simple web apps more complex than with the PaaS alternative, however it does provide the added benefit of flexibility - like with the onset of OpenStack, and open source cloud in general..

The right way to think of PaaS and Docker

In my opinion, the right way to think of the two is in a layered approach in which both the PaaS containers and Docker fit together as outlined in the diagram below.

As seen in the diagram above,  PaaS becomes one use case for deploying simple web apps. For more complex application deployments, it makes more sense to use Docker directly. In this context orchestration becomes a central piece for managing both use cases. Orchestration also provides a layer of abstraction between the application components and the underlying infrastructure.

The important realization is that in this context the role of PaaS as we know it becomes more of a niche role, yet still remains a fairly common and important use case. It is also important to note that the two models are not mutually exclusive. An application that runs on a PaaS container could interact with services that are managed directly through a Docker container, just as you would use other cloud services such as database as a service, load balancer as a service or any other on-demand resource.

Another important note is that the combination of Docker and orchestration opens a new path to manage dynamic workloads not just on an IaaS environment, but also directly on bare metal. I’m starting to see more use cases where this option becomes more popular, specifically for managing dev and testing environments where the use of a full blown cloud infrastructure may just  be overkill.

Final Note

When we started the journey into the cloud world a few years ago, the underlying cloud infrastructure was very complicated. As a result, the only way possible to simplify the application deployment experience was to take extreme measures in the level of abstraction, as well as by forcing an opinionated architecture to ensure that the application fits into typical cloud architecture. As the underlying infrastructure and tooling evolves,  those extreme measures are no longer needed for all use cases. What we’re now seeing is basically a shift of of pieces in the PaaS architecture that was previously embedded within the "PaaS Box" such as containers and orchestration that have becomes more generic, along with independent services that have their own right to exist.

The good news in all this, is that as users we are finally beginning to receive the simplicity of PaaS, that was previously limited only to greenfield web applications, for use with a  much broader spectrum of applications and architecture styles.

So to answer the question from the opening note, do i need PaaS if i use Docker? The answer is, yes. It makes sense to have the two combined, but the role of PaaS as we know it has narrowed down significantly.

The VMware OpenStack Cloud Conundrum

OpenStack is an open source cloud infrastructure that is considered by many as a cost-effective alternative to VMware.

In reality, the transition from VMware to OpenStack isn’t that trivial, which leads most enterprises to take a hybrid cloud approach. This means they run their OpenStack infrastructure alongside their existing VMware infrastructure with an aim to gradually transition workloads into their new OpenStack environment.

Recently, VMware announced their own VMware Integrated OpenStack, which is a VMware-supported OpenStack distribution providing tighter integration between existing VMware environments and OpenStack.

In this post I will discuss three of the options for putting OpenStack and VMware together and weigh what I believe are the pros and cons of each approach.

1. Using OpenStack with VMware Hypervisor plug-in

OpenStack Nova comes with a pluggable architecture for integrating various hypervisors. It supports both KVM, VMware as well as Hyper-V.

The first option for integrating VMware into OpenStack is through the ESXDriver for Nova.

In this option the Nova scheduler can spawn VMware ESX VM’s through an ESX enabled node (as opposed to KVM which runs directly from the Nova compute node).

Pros:

In this approach, we can potentially re-use our VMware images assets and easily import them into our OpenStack environment.

Cons:

1. Limited use of VMware features:  ESXDriver cannot use many of the vSphere platform advanced capabilities, namely vMotion, high availability, and Dynamic Resource Scheduler (DRS).

2. Limited Portability: There are some features of VMware, such as vMotion, that many enterprises rely on today and would need to be turned off as we move to OpenStack. As some of the images were built with the assumption that this unique VMware features exists, it wouldn’t be possible to transition them into an OpenStack environment and expect them to work.

In summary, while this option makes sense, the cost and technical limitations make it less popular. In fact, a recent OpenStack survey report indicates a fairly small percentage of users who actually use this feature.

Using VMware vSphere with OpenStack

In this approach we utilize the compute resources using vSphere. This will allow us to take advantage of all the VMware features that come with vSphere and overcome the limitations mentioned above.

In this case, the entire vCenter ESX cluster appears as a one big hypervisor. The actual allocation of the ESX hosts is done through VMware vCenter and is not exposed to the Nova Controller, as outlined in the diagram below:

Similarly, we will plug-in the VMware Storage and Network services to allow for even more complete integration across the compute, network and storage stack as outlined in the diagram below:

You can learn more about how this integration works in the OpenStack reference guide.

Pros:

The main advantage of this approach is that it enables users of VMware to benefit from both worlds; on one hand, they can use OpenStack as an open API and on the other hand they can utilize their existing VMware infrastructure.

Cons:

The main disadvantage of this approach is that it creates a completely different OpenStack implementation, which has some serious differences in its implementation and behavior from the original open source version of OpenStack - specifically in the way the compute nodes are managed.

The Elephant in the Room:

One of the main motivations to transition to OpenStack in the first place is to cut costs.

In both options we rely on the VMware stack, and therefore the actual savings are still unknown.

3. Using a Common Management and Orchestration as an abstraction to both VMware and OpenStack

In the third option we will not use any of the OpenStack VMware plug-ins, but instead we will use an orchestration layer as higher level abstraction between OpenStack and our VMware environment.

The orchestration layer provides a common management and deployment infrastructure. In this approach we will not be trying to force the VMware infrastructure to fit into the OpenStack API, but instead we will just map the different calls to either OpenStack or VMware into the appropriate type. In this way, the application is kept aware of whether it is running on OpenStack or VMware. However, since the calls to each of the infrastructure components are now centralized into one driver per environment, it is managed once for all the applications. Additionally, there is a default implementation for the built-in types, so in most cases the user will need to deal with the details of implementations of each element type only for specific customization.

Pros:

1. We are able to utilize all of the features and capabilities of each infrastructure with no limits.
2. We reduce dependency risk - With this abstraction we’re less vulnerable to a specific infrastructure and we keep our options open to move or add new environments as needed.
3. Support for vSphere and vCloud -Since we are not limited to the use of a specific API, we can integrate with both vSphere and vCloud API.

Cons

1. We are shifting the dependency on the management and orchestration layer.
2. We may lose some of the management and orchestration capabilities that are specific to each of the environments.
3. Additional customization effort: Since we’re not relying on a common API, we may need to customize the built-in types per environment to fit our specific needs. Having said that, it is important to note that this customization is only done once for all of our applications. Additionally, it is expected that over time the default types will cover most of the use cases. Therefore, the need for customization will be minimized.

My Take

Given that the cloud infrastructure world keeps on changing and evolving very rapidly (not just between OpenStack and VMware), any tight integration approach will have a higher chance of breaking compatibility or being limited to the least common denominator at some point. We also need to be aware that even though in the first two options we remain compatible with the OpenStack API, we still end up with different OpenStack implementations from a behavior perspective.

On top of that, we need to be prepared for new disruptions. This is actually taking place right now, for example, with Docker orchestration that continues to disrupt and challenge the way we handle our compute and network infrastructure.

With this in mind it would be too risky not to keep our options open.

Having an abstraction layer at a higher level of the stack gives us the benefit of being less vulnerable to changes at the lower level infrastructure. It also provides us the flexibility to adopt new infrastructure changes in the future.

Having said all this, the question “Are we minimizing or just shifting the risk?” still lingers.

This is where TOSCA comes in handy. TOSCA provides a standard way to describe our application blueprint. This significantly reduces our dependency on a specific implementation of the orchestration and management layer and this was one of the main reasons that led us to choose TOSCA when we designed the third generation of our orchestration with Cloudify.

References

• OpenStack VMware integration

• http://www.slideshare.net/danwent/hk-2013-v-mware-session?related=1

• OpenStack Hypervisor support Matrix

• OPENSTACK COMPUTE FOR VSPHERE ADMINS (highly recommended and the source for many of the diagrams in this article)

• Cloudify - Tosca Based Orchestration

Do I need OpenStack if I use Docker?

 

Docker has broken a record in the speed in which it moved from being a disruptive technology to a commodity. The speed of adoption and popularity of Docker brings with it lots of confusion.

In this post I wanted to focus on a trend of commentary that has been gaining popularity, that I’ve started to hear more often recently from users who just started using Docker.  Whether it makes sense to use OpenStack if they’ve already chosen to use Docker.

Before I give my take on this, I wanted to start with a short background for the rationale behind this question.

Background

In its simplest form, Docker provides a container for managing software workloads on shared infrastructure, all while keeping them isolated from one another. Virtual machines such as KVM do a similar job by creating a complete operating system stack of all the OS devices (through a Hypervisor). However, unlike the virtual machine approach, Docker relies on a built-in feature of the Linux operating system named LXC (Linux Containers). LXC utilizes the built-in operating system features of process isolation for memory, and to a lesser degree, CPU and networking resources. Docker images do not require a complete boot of a new operating system, and as a result, provide a much lighter alternative for packaging and running applications on shared compute resources. In addition, it allows direct access to the device drivers which makes I/O operations faster than with a hypervisor approach. The latter makes it possible to use Docker directly on bare metal which, often times causes people to ask whether the use of a cloud such as OpenStack is really necessary if they’re already using Docker.

This performance difference between Docker and a hypervisor such as KVM is backed by a recent benchmark done by Boden Russell and presented during the recent DockerCon event.

The benchmark is fairly detailed, and as expected, it shows a significant difference between the time it takes to boot a KVM hypervisor to a Docker container. It also indicates a fairly big difference in the memory and CPU utilization between the two, as can be seen in the diagram below.

This difference in performance can map to a difference in density and overall utilization difference between the two in similar proportion. This difference can easily map into a big difference in cost that is directly affected by the number of resources needed to run a given workload.

My Take:

• This question has nothing specific to do with OpenStack and can be applied similarly to any other cloud infrastructure. The reason it is often brought up in the context of OpenStack, in my opinion, is due to the fact that OpenStack is fairly popular in private cloud environments which is the only environment in which we can even consider a pure Docker alternative.

• It's about the hypervisor stupid:

Many of the performance benchmarks compare Docker versus KVM, and have little to do with OpenStack. In fact, this specific benchmark of both KVM images and Docker containers was running through OpenStack which shows that the two technologies works nicely together. In that context, most of the utilization  arguments become irrelevant when I choose to run OpenStack on top of a Docker based Nova stack as illustrated in the diagram below taken from the OpenStack documentation.

• Cloud infrastructure provides a complete data center management solution in which containers or hypervisors, for that matter, are only part of a much bigger system.  Cloud infrastructure such as OpenStack includes multi-tenant security and isolation, management and monitoring, storage and networking and more. All of those services are needed for any cloud / data center management and have little dependency on whether Docker or KVM are being used.

• Docker isn't (yet) a full-featured VM and has some serious limitations like security, Windows support (as indicated in the following email thread in the Red Hat mailing list), and therefore cannot be considered a complete alternative to KVM just yet. While there’s ongoing work to bridge those gaps, it is safe to assume that adding the missing functionality may come at an additional  performance cost.

• There’s a big difference between raw hypervisor performance/containerization and application performance, as indicated in the following graphs from the benchmark results. A possible explanation for that is that applications often use the caching technique to reduce the I/O overhead.

• If we package Docker containers within a KVM image the difference can become negligible. This architecture often uses hypervisors for managing the cloud compute resources, and an orchestration layer such as Heat, Cloudify, or Kubernetes on top to manage containers within the hypervisor resources.

Conclusion

This brings me to the conclusion that the right way to look at OpenStack, KVM and Docker, is as a complementary stack in which OpenStack plays the role of the overall data center management. KVM as the multi-tenant compute resource management, and Docker containers as the application deployment package. 

In this context a common model would be to use Docker for the following roles:

• Docker provides deterministic software packaging and fits nicely with the immutable infrastructure model.

• Docker for excellent great containerization of microservices pods

• Using Docker on top of OpenStack as well as a bare metal environment

Having said all of the above, I do see cases mostly for well defined workloads, where the use of cloud infrastructure isn't mandatory. For example, if I were to consider the automation of a small shop development and testing environment for DevOps purposes, I would consider using Docker directly on a bare metal environment.

Orchestration can be a great abstraction tool between the two environments

One of the benefits of using an orchestration framework with Docker, is that it can allow us to switch between OpenStack or bare metal environments at any given point in time. In this way, we can choose either options just by pointing our orchestration engine to the target environment of choice. OpenStack Orchestration (Heat) declared support for orchestrating Docker starting from the Icehouse release. Cloudify is an open source TOSCA based orchestration that works on OpenStack and other clouds such VMware, AWS and bare metal, and recently included Docker orchestration .  Google Kubernetes is associated mostly with GCE but could be customized to work with other cloud or environments.

References:

• Using Docker as and OpenStack compute resource (Nova)

• Orchestrating Docker containers through OpenStack Heat

• Orchestrating Docker containers on OpenStack, VMware, AWS, bare metal and Other clouds through TOSCA/Cloudify

• Docker limitations as indicated in Red Hat mailing list

• The boring truth: full virtualization and containerization both have their place

• Using Docker as immutable infrastructure

• Docker drops LXC as default execution environment