Nati Shalom's Blog

The Interactive Cloud (Part II - Adding Operational Awareness to your Application)

In part I of of this topic, I covered the general concept and motivation behind the "Interactive Cloud". I described how the interactive cloud enables applications to be managed in a similar way to the way we run our business. In this post, I discuss further details of what it means to add operational awareness to the application. To do that, I refer to our specific experience in GigaSpaces. Towards the end of the post, I point out specific code snippets as an illustration of that model.

Background

GigaSpaces 7.0 Application Cluster Management API is an API for managing application clusters, and for giving applications operational awareness. The reason why we came up with this API in the first place, is quite interesting. When we started GigaSpaces, a typical deployment was primary-backup with multiple clients connected to this pair. Sooner then we anticipated, we found ourselves supporting cluster deployments of 10s and 100s of nodes per application. We are now at a point where customers are starting to use GigaSpaces XAP as a service shared between multiple applications, which in itself increases the deployment size quite significantly.

In this type of environment, troubleshooting a production system can be a real nightmare for everyone involved. We started to add all sorts of metrics and logging information into our system but it hardly improved things. The process of finding the right information at the level we needed was still a heavy-duty task.

When we looked back and tried to analyze how we learn about the application’s behavior, we saw that quite often we identified the time when things started to go wrong, and then, using the timestamps on the different operations, ran a complex correlation process to see what happened on each machine during that time. In many cases when we hit the first area of trouble, we were still missing information, because we were running without fine-grained logging, for performance reasons. So then we would turn up the log level and wait for the next event, and so forth.

We realized that just adding more probes and log levels and using some sort of CEP (Complex Event Processing) to parse all the garbage we were going to generate, on a continuous basis, was not going to be good enough. Even worse, if the reporting system got so complex, we would eventually end up going back to the old logs because the reporting system would inevitably lose some of our events :)

We figured that we could not afford to continue down that path – the cost, both on our end and on the customer’s end, was way too high. We needed to try a totally new approach.

The Ideal Solution – Getting Help from the Applications themselves

When we analyzed what would be the ideal process, we came to the realization that the application should be responsible for the monitoring. When the application starts to behave abnormally (for example, CPU spikes, GC spikes), or experiences network failure, it should do two things:

1. Take corrective action – the application should figure out if the problem is that resources have been exhausted. If that is the case, it should try to rebalance itself to reduce the load automatically.
2. Provide information to help analyze the root cause of the event that occurred – the application should point out the machines that are suspects and ask those server instances to increase their logging level and print a snapshot of thread dump. This provides only the information that is actually needed to perform root cause analysis, and at the right level of granularity.

It wasn’t surprising that some of the root cause analysis indicated that one of the biggest contributors to failures, is human error or misconfiguration. That led us to the realization that the way to reduce those factors would be through complete automation of manual deployment and maintenance processes.

Another thing we realized during that process, is that when things went wrong, quite often the operation guys became fairly useless at a fairly early stage of the process, and the application guys had to be called in to understand what happened. Not surprisingly, the application guys didn’t even know how to get the required information or didn’t have permission to get that information, and had to rely on someone else to do that for them – which made the entire process really long and complex.

All of this led us to the realization that we needed to create an application cluster management and monitoring tool that could be used by both the application developers and the operation guys. The application guys could use that API as part of their development cycle, and the operation guys could use it to receive the right level of information into their management and control dashboards.

In the following section, I provide a short sneak peek into this API, and how it is structured.

XAP’s New Application Cluster Management and Monitoring API (a.k.a. the Administration API)

There are basically two ways to design  the API for this challenge: define the interaction model and drive the API from there, or define the domain model and drive the interaction model from the domain model. We chose the latter.

1. Defining the Application Management Domain Model

The first step was to define the domain model of our application management. The diagram below maps out the “nervous system” of an application running on XAP – it shows the core layers of the application management domain model.

 

2. Defining the Interaction Model – the API

Once we mapped the nerves and sensors of our application, we could start to wire them. This enabled us not only to monitor the state of the application components, but also to interact with them when we needed to take corrective actions (see snippet below).

3. Making Operational Awareness an Integral Part of the Development Framework

If we want the application to take more responsibility for its operational behavior, we can’t just point it to an external management and monitoring tool that doesn’t speak the “developer language”.  We need to make the proactive and reporting behavior an integral part of the development framework. This is illustrated in the examples below.

Application Code that uses the Cluster Management API - a Few Examples

Below are a few examples that show how applications interact with the new Administration API, using a groovy-based API. These snippets were written in Groovy but they can be be called in Java as well.

Example 1 – Getting Statistics

This example uses Groovy closures to get callbacks when a new container or machine joins the network, and when a specific application event happens.

Example 2 – Calculating Average Load on a Web Container

This code snippet shows how to get an aggregated result of the total number of requests/sec from all the web containers in the cluster.

Example 3 – Scaling Up:

The code snippet below shows the process of scaling out, when the number of requests exceeds the maximum number of requests allowed per instance.

Note that the call to incrementInstance() generates an entire match-making process behind the scene. What happens is that a new web container is spawned, but the provisioning of this new container is based on the SLA that was defined at deployment time. In our case, it is deployed only on machines that don’t already run a web container and have spare CPU and memory capacity.

Initial Success Stories

Since we came up with the API, we started to see its use spreading in areas we didn’t even think of. Here are a few examples:

• Helping customers reproduce problem scenarios – now that we can script almost every part of the application, we can easily script even very complex scenarios, where the application starts to execute a few things and then crashes only when a certain operation happens, etc. The fact that we can script these scenarios makes it easier to create a reproduction scenario, and from there, the road to spotting the problem and fixing it becomes significantly shorter (see detailed reference here).
• Third part integration – one of our partners created a fairly extensible Eclipse plug-in. The reason he was able to do that pretty much by himself, was because the API gave him everything he needed in order to control the application development and testing lifecycle (see more details in Jeroen Remmerswaal post here).
• Writing a complex SLA – we were able to easily write complex SLAs, for example, automating deployment of applications across multiple data centers and making sure that primaries and backups never run on the same data center. All that without any human intervention.

What’s Coming Next?

One of the immediate things we are going to enhance, is to provide more built-in SLAs, and a set of utilities that perform deployment and implementation of mainstream scenarios, such as auto-scaling of web applications and auto re-balancing of data grid deployments. This is going to serve as a cornerstone of our new cloud-enabled middleware initiative, where all the classic, enterprise-grade middleware components become available as a service. In fact, some of our customers are already using our In-Memory Data Grid as a shared service between multiple applications within the organization. The idea is to make that type of deployment simpler and intuitive.

Take Part in this New Initiative

I can go on with more of my thoughts on what should be done, but I’d rather stop here and ask all of you for your feedback. If you’ve faced similar challenges, how do you see this solution? What would you want to see in this new package?

For those who want to try out this new API, it is worth noting that it is actually provided for free as part of our Community Edition, so anyone can simply download it and try out the examples from this post.

The Interactive Cloud (Part I)

When we think about cloud computing, the first thing that comes to mind is on-demand computing. Indeed, cloud computing and virtualization providers made it easy to get a machine by just providing a credit card and making an API call. However, that being said, what happens after you get that pool of machines? How do you deploy the application into these machines? How do you know how the application behaves? How can you react when something breaks or when the load on the system gets too high?

In the human world, many organizations face pretty much the same challenges. It is fairly easy to get a good infrastructure for business – office space, electricity, telephones, computers – but as a manager, it’s really difficult to place the right people in those offices, give them sufficient work but not so much that they’re totally overloaded, track what they’re doing, and react properly when something goes wrong. A recent and very successful methodology for dealing with this challenge, which is quickly gaining popularity, is called Agile, Scrum,  or Lean. The common ground of this family of methodologies is that unlike the centralized management approach, we work through short iterations and continuously correct the course of our actions based on the actual progress. We rely heavily on the individuals to take responsibility for their actions and be part of the decision-making process. We assume that those individuals will take their own corrective actions for things that fall under their direct responsibility. The program manager is only responsible to make sure that the course (and budget) is kept on the right target and shift resources or allocate new ones if needed.

Today’s Operations and Applications – Just Like Centralized Management

Chances are that you are aware of the Agile methodology and recognize that it solves many of the huge difficulties of traditional, centralized management. But at the same time, if you think about it, you are still running your applications and IT operations according to the defunct, centralized management paradigm. We’ve invested years in creating an artificial separation between what is known as “operations” and “applications”. In this way of thinking, “applications” is responsible for maintaining a certain business logic and “operations” is responsible for monitoring and maintaining the application. Operations management is responsible for monitoring the CPU, memory and every individual aspect of the network configuration, application configuration, and consumption of resources. The application is not really aware of any of this, and usually you rely on the operations guys to take corrective action, or at least give you a call and let you know, when something out of the ordinary happens.

This seems natural because it’s a pattern we’ve all become accustomed to. But just imagine managing a real life project (with people) using this same approach. It would take the centralized management approach to frightening extremes. Here’s how it would work – there is the project manager (operations) and the employees (application), with absolutely no direct communication between them. Each employee has a set of sensors attached to her, which report on health conditions (heartbeat, blood pressure, and so on), and another set of sensors that report how quickly she is working. The project manager views all this data in a nice dashboard with nice graphs and all sorts of reporting schemes. Whenever there is an alert – for example, one of the employees is running a fever, or is working very slowly for some reason, the project manager troubleshoots the issue and sees how to solve it and whether it affects the project.

That’s ridiculous, isn’t it!?

In real life, especially when working with Agile, employees are responsible for their output. If they don’t feel well, or are working slower than usual, only the individual employee can really know what the problem is – or whether there is a problem at all. They take a rest or figure out what else they can do about it. They only report back to the project manager if they really can’t deal with the issue themselves and it’s starting to affect their ability to meet the project goals. The manager doesn’t need to constantly monitor their health or throughput – the only thing worth monitoring is if they can or can’t do their job. The reason is less of interest. More importantly, the manager relies on the employees to report when such an event happens, and to take corrective action themselves.

If that’s so obvious, why can’t we apply the same same common sense to the way we run our software? Why can’t we make our applications aware of their condition and able to take some responsibility for themselves? This is important in any computing environment, but I would argue that it is especially important on a cloud or in any virtualized environment, in which the operations/application paradigm totally breaks down.

The Solution: Bringing Operational Awareness to your Applications

Like a real-life employee, the application needs to be responsible for its own health. It needs to report only on general status and any shortage of resources. The operation guys don’t need to continuously monitor the health condition of each individual application component. They should rely on the applications themselves to send the right warning signals before something breaks, and suggest corrective actions in a language that the operation guys can understand. For example: “I need X amount of additional resources ” or “I can release Y amount of resources after I've done my duty.” Deciding which specific task needs to be assigned to each of the available resources should be the application’s responsibility – in any case, it probably knows best how to utilize these resources. 

To make this a reality, you need to add operational awareness to your applications. By operational awareness I mean that the application needs to be written with the assumption that it can be asked to be moved at any point in time, and the application needs to take responsibility for events that can be solved at the application level, and take corrective actions to meet its SLA. The manager should only be notified in cases where the application has exhausted its physical resources and there are no other available resources. For example, if one of the instances of the application runs out of memory or needs more CPU power, and there are spare resources that are not utilized by other resources within the application, it is the application’s responsibility to balance its resources within its own realm. But even if there are no available resources, the manager doesn’t need to bother with the fine-grained details of how to balance the individual application resources. Balancing these resources should be kept within the scope of the application itself. The manager should only bother to allocate the pool of resources between the various applications.

Learning from Amazon EC2

A working example of this approach is the Amazon EC2 API. In my view, one of the revolutionary concepts that made EC2  so successful, was the realization that rather then creating high level tools for managing the data center, Amazon chose to provide an API and let the application developers figure out how to run their application. The API opened up a whole ecosystem of third party solutions, and a whole new market. What’s interesting is that in EC2, when I want to auto-scale my application, I don’t call the operations guys to do that. There is a very slim chance that someone who doesn’t understand my application would be able to write such a policy without breaking the application. Most of the applications that run on EC2 just call the EC2 API when they want to spawn a new machine or manage their high availability, or auto-scaling of their application. The nice thing about the API is that it doesn’t prevent Amazon and others coming up with more high-level tools that are much more useful for the operation guys at a later stage. The main difference is that the application takes more responsibility for ensuring that it meets its SLA, and the operations guys get the type of information that makes sense to them, and more importantly, something that they can easily act upon and take the appropriate corrective actions.

The Need for an Application Cluster Management API

Amazon provides a comprehensive set of APIs that work at the Virtual Machine level. Back to our real-life project analogy, the Amazon tool provides us with the real-estate  - the office space, the network, the computers. What we don’t get is the access to the individuals that do the actual work. For this reason, keeping the API at the virtual machine level is often not granular enough, as it doesn’t enable us to interact with the individual application components, manage and automate the application’s deployment lifecycle, define its dependencies and distribution model. There are other frameworks such as Capistrano that became popular, mostly in managing Ruby deployments. This type of tool is becoming popular in other cloud deployment settings as well. Capistrano can address the application deployment and basic life cycle management, and provide a good foundation for automation of those processes.

While the combination of the two is a good starting point, it is lacking the notion of application cluster. In an application cluster, you want to have more explicit semantic support for managing deployment topologies (partitioning, replicated). You need built-in support for defining backup and primary, and you need to be aware of the data affinity requirement of certain elements in the architecture. Without all this, enabling the application to re-balance its resources when a new machine joins or leaves the network, is close to impossible.

The Interactive Cloud

The combination of image-level provisioning and an application cluster management API, leads to a new form of cloud deployment experience that I refer to as the interactive cloud.

The interactive cloud operates fairly closely to the way we manage human-led projects. The image API is equivalent to the physical resources that are critical to run our project, e.g. the office, the computer, and the network. Another part of the project is managing the individuals that run the project. Learning the right way to manage our teams, from the lessons of Agile project management, is through a close iterative process. This is where the Application Cluster Administration API comes into play. This API gives the application developer the ability to control and automate application deployment, and continuously balance the way the application runs, to ensure that it meets its required SLA. It also provides a callback mechanism when you run out of resources and when some of the idle resources can be released. You can also easily generate an aggregated view for operations (without going down to the level of what each individual component did at each given point in time).

With this combination, you can spawn new applications, automate their deployment and lifecycle, start new machines as needed and keep a close watch on your application behavior after it has been deployed. When a new machine joins the cluster, the application automatically rebalances its resources to take advantage of that new machine, and when machines fail or are removed from the cluster, the application rebalances itself on the smaller resource pool. If something breaks, the application automatically scales down, allowing it to continue working despite the failure, and in the background allocate an alternate resource to fit in instead of the new one.

IMO, a cloud/virtualized environment forces us to go in this direction. This is an area where the old way of separating “applications” from “operations” is just not going to cut it.

Final Words

There is nothing really new about the solution I propose here, and the concept behind it. Quite the contrary. The interactive cloud allows you to apply proven methodologies that work well for many critical projects in the real world, and apply them to the way you run your applications. Most people who run their application in the cloud have already started to build and manage their applications in this way, using tools like the Amazon EC2 API and Capistrano. However, this was mostly done as an afterthought and was mainly driven by intuition and common sense. My attempt in this post is to back up these initial efforts with a clear methodology.

In my next post (Adding operational awareness to your application), I will use the Administration and Monitoring API, new in GigaSpaces XAP 7.0, as an example for such a framework. It is worth noting at this point that this API is actually provided for free as part of our Community Edition, so anyone can simply download it and try out the examples I will provide in the next post. I’m looking forward to showing that this vision of the interactive cloud is really very tangible and is already at everyone’s fingertips.