Nati Shalom's Blog

Java and Center Stage: Meet us at JavaOne 2011

Next week, I’m going to be in JavaOne. Unlike last year, where I expressed lots of skepticism about the way Oracle drives the Java Community, it seems that this year things are starting to get back on course – not that the skepticism has vanished completely, but at least there is a stronger sentiment that things are starting to settle down and take a more positive course.

It seems that now Java is taking center stage in the Cloud world, with more application platforms such as SalesForce/Heroku, VMware, Redhat/OpenShift, Cloud.com, JClouds, and obviously GigaSpaces providing a rich set of offerings that are based on Java.

One of the most interesting things in my opinion is that the combination of Cloud and Java brings new opportunities that until recently were unheard of:

 

• Heroku, which was very much Ruby-centric, added support for Java.
• GigaSpaces recently announced our partnership with Microsoft to bring Java to Microsoft's Azure community.

What’s interesting in those two examples is that the Cloud forces us to break away from the language wall. It brings communities that were living in completely isolated silos much closer together.

This forces a very different way of thinking about Java - and about the next generation Java platform specifically.

The new platform should be:

• Open for other languages (through integration points like Protobuff, Thrift, REST, and more, as well as the ability to run those languages)
• Embrace NoSQL implementations as an integral part of the platform
• Add support for dynamic data structures such as JSON, and schema-free document models
• Bring Dev and Ops closer by providing better exposure of operational aspect as part of the development frameowork
• Make cloud deployment a first class citizen

It seems that Java EE 7 is taking the right path in that direction. From Java EE 7 and Cloud Computing:

• The Java EE platform architecture is taking into account the operational impact of the cloud, more specifically by defining new roles, such as a PaaS administrator.
• The Java EE platform may also establish a set of constraints for PaaS-specific features, such as multi-tenancy, that deployments may have to obey. Applications may also be able to identify themselves as designed for cloud environments.
• All resource manager-related APIs, such as JPA, JDBC and JMS, will be updated to enable multi-tenancy. The programming model may be refined as well by introducing connectionless versions of the major APIs.
• Java EE will define a descriptor for application metadata to enable developers to describe certain characteristics of their applications that are essential for the purpose of running them in a PaaS environment. These may include being multitenancy-enabled, resources sharing, quality of service information, dependencies between applications.

What This Means

Java is regaining some of its luster from the last few years. It was being perceived as vaguely pedestrian and ordinary, and languages like Ruby and Python were being seen as more dynamic environments for rapid and scalable deployments.

Now we're seeing Java as the lingua franca of enterprise development. With so many environments being heterogeneous, a homogenous platform becomes a very desirable resource, and the JVM is able to run many languages in an integrated environment: Java, Ruby, Python, Scala, and Groovy, to name only a few, and many of these can run as compiled code and as scripts.

Even beyond the multiple programming language support of the JVM, you have very broad support for almost any number of remote procedure call mechanism you can think of. The old J2EE approach, where you defined your requirement and there was a specific way to fulfill it (which meant that every architecture was more or less nudged to the middle), is no longer a real limit.

If you can do it, you can do it in Java, without Java forcing you to sacrifice in the process.

This is a very powerful concept.

It puts your organization back under your control. This is desirable because you know your product and architecture better than a community process does. You know whether you need to support REST here, and SOAP there; you know you need to use NoSQL in this area and MySQL in that area.

A Remaining Challenge

Java EE 7 may be becoming aware of the new cloud-based environment, but it's still only one aspect of application deployment and design. What it does not address is the entire application environment.

This is where applications such as Cloudify come in. Cloudify treats every artifact involved with your deployment as a managed element.

Consider: when you deploy an enterprise application, you're not just deploying some operational code. That code has external requirements, like a database (or NoSQL warehouse); it might also rely on an Apache-based load balancer, or perhaps another independently-deployed artifact.

That's quite a burden on operations, because all of these are separately deployed and managed.

Cloudify, however, centralizes the management of each element your application is composed of, over the entire application's lifecycle, and provides a bridge to whatever environment you wish to use.

For example, you could have a MySQL database, used by a Java application hosted by a single Tomcat instance, fronted by an Apache httpd server, hosted on an internal Linux server. Cloudify can deploy and monitor each of those elements: the MySQL database, the schema and initial data; Tomcat; the Java application; Apache httpd, and the server upon which it all runs.

Now let's look at a larger deployment: a set of MySQL servers, a cluster of GigaSpaces XAP nodes, an instance of GlassFish hosting some web services, six .Net-based web services, and five httpd servers being load-balanced, deployed on a platform such as Microsoft Azure. From Cloudify's perspective, the deployment and management process is the same. You have more artifacts to deploy, and you're telling Cloudify to deploy on a different platform... that's it.

Cloudify can make sure each resource is running optimally, as well. If you can measure an attribute - such as CPU load, or disk space, or free memory - you can tell Cloudify to assert that some headroom exists, and what to do if a metric is exceeded.

For example, if you have a CPU running at 100% for an extended period of time, Cloudify can automatically deploy another instance of the component consuming the CPU, and configure load balancing so your application is more scalable, without human intervention.

Likewise, if you have four servers running a given process, all at 0% CPU load, you can reduce the number of containers with that component so that you're not overallocating resources, saving energy and money.

The key thought here:

You know how to solve your problem. We can help.

We'll be on the floor at JavaOne 2011, in booth #5002. We'll be happy to show you both XAP, our industry-leading application platform, and Cloudify, our deployment solution for the cloud-enabled world.

 

 

(This post was co-written with Joseph Ottinger, who unfortunately won't be able to make it to JavaOne this year. But he says that the best people in GigaSpaces will be there, so you're not missing out!)

Multi-tenancy: does it have to be that hard?

Multi-tenancy is a term that is used often in the context of SaaS applications. In general, multi-tenancy refers to the ability to run multiple users of an application on a shared infrastructure. The main motivation for doing this is efficiency, or in other words -- reducing the cost per user in comparison to a dedicated system where each user has their own dedicated environment.

Over the past few months I've had discussions with various SaaS providers, in which I tried to learn about the main challenges that SaaS providers face when they deal with multi-tenancy. As it happens, I came across an interesting article on this very topic, Why Multi-tenancy Matters in the Cloud by Alok Misra. The article later sparked an interesting discussion thread on the cloud mailing list, which I also find to be a good reference for diverse thoughts on this subject.

In this post I want to summarize some of my findings on multi-tenancy, as well as suggest a model that can make multi-tenancy significantly simpler.

The common challenge: Multi-tenancy at the data layer

There are many approaches and levels of multi-tenancy, depending on the application layer and type of application you're dealing with. For example, trying to separate user accounts in a CRM application is quite different than separating various batch processing jobs in a risk management application. That said, what seems a common challenge in almost every SaaS application is multi-tenancy at the data layer. To put it simply, the challenge is how to share data resources (database in most cases) among multiple users, while at the same time ensuring data isolation between those users, as if they are running on completely physically seperated servers.

In the remaining part of this post I'll address how multi-tenancy is currently being dealt with at the data layer, and how the existing challenges can be met.

How is multi-tenancy being done today?

For this part of the story, I think it might be useful to first look at the two extreme approaches: The fine-grained approach and the cross-grained approach.

Fine-grain multi-tenancy (Salesforce.com model)

For many SaaS providers the “poster child” of multi-tenancy is Salesforce.com. There is a very useful videocast explaining the Salesforce approach to multi-tenancy. In a nutshell, their approach is to share the same database instance between different users, and define the data model so that each table has a primary key identifying the customer ID. In addition, there is a separate metadata table that translates every query into a user-specific query before the query hits the database.

• Pros:
  • Very fine-grained multi-tenancy resulting in optimum cost efficiency per user.
• Cons
  • Complexity: Requires changes in the data model.
  • Doesn’t fit to existing application: Existing applications require a complete rewrite.
  • Modest degree of isolation: Users sharing the same resource can impact one another under heavy load situations. A maintenance/patch release required for a particular user might affect other users. In a more extreme scenario, there is also exposure to a higher degree of vulnerability. For example, a bug in the query mapping could point one user to data belonging to other users just by picking up the wrong customer ID.

Cross-grain multi-tenancy (hosted service model)

Most hosting providers (including hosted IT providers) enable outsourcing of certain IT applications. Efficiency is gained by sharing maintenance overhead, labor costs, and other data center costs such as network, electricity, and more.

• Pros:
  • No code changes are required, resulting in greater simplicity.
  • High degree of isolation.
• Cons
  • Lower efficiency and therefore limited cost margins.

Challenges with today's approach to multi-tenancy

The challenge with the fine-grained model is that it is fairly complex to maintain and implement. If you have an existing application, it requires a complete rewrite and also forces fairly significant changes in your existing data model. The challenge with the cross-grained model is that cost margins per user are still rather high. In a competitive environment, even a single competitor offering similar servicse to yours using the fine-grained approach could easily put you out of business.

The elasticity challenge

The other challenge that neither approaches addresses today is dynamic elasticity. What happens when a customer grows beyond their allocated shared capacity? Both approaches seem to rely on certain capacity assumptions, and can scale by splitting multiple users between their shared resources. However, none of the approaches seem to handle a situation where a particular user's capacity grows beyond the allocated resources.

Jayarama Shenoy provides a good introduction to that challenge in his comment in the discussion thread:

So, how does elasticity work in a multi-tenancy case? Say 100 tenants share an app on a server (by which I would think they go all the way down to a shared database). Then,say, five of them grow quite rapidly such that they need to be moved off that server, and the other 95 can continue to exist on the original server.
So 'something' in the application has to know how to pick off the records that belong to these five, copy it off into a second instance of that database and kill, move & restore application service for those tenants. (Of course, without being privy to the data contained in those records). And of course there's an implied load balancer element in front of all this to keep up appearances to the clients.

Based on this analysis I believe that the main challenge with the current approaches is that we have too few options available. No option seems to provide a good solution to the elasticity challenge, and this is what is making multi-tenancy “so damn hard”. This is especially true for existing businesses with existing products.

The solution: Making multitenancy simple

Learning from the past operating system experience

Some of us probably remember the days when Bill Gates came up with DOS, which was basically a single-user system. At the time, if you wanted to provide a multi-user application you had to write it yourself, and sometimes that required going down to the core interrupt level and program in assembly language. This made multi-user systems on DOS very uncommon. Unix, on the hand, was built for multitasking, multi-user environments, and therefore it was quite common to build multi-user systems in the Unix environment. Later on, when Windows NT came out, all that changed. Today we can’t even imagine a scenario where we would build an application that wouldn’t support multiple users. 

Lesson learned:

As long as the underlying infrastructure (the operating system in the example, and the database in our specific discussion) is not built for multi-tenancy, trying to solve the multi-tenancy challenge at the application level is doomed to be as complex as writing a multi-user system on DOS. 

In fact, once multi-tenancy was dealt with properly at the operating system level, writing a multi-tenant application became significantly simpler and therefore widely adopted. The same should apply to databases as well.

The database is the problem!

Almost all of the attempts I've seen in both the Salesforce approach and the hosted environment approach were built under the assumption that the database is a fixed “concrete” construct that cannnot be broken down easily into “smaller bricks”, let alone scale dynamically. It is therefore not surprising that most of the attempts to deal with multi-tenancy have tried to address the challenge outside the database.

Now, imagine what your approach to the challenge would be if a database could be easily broken down into small “bricks” without huge overhead, and if it could scale dynamically?

Multi-tenant data service

Similar to the experience we had with operating system, I believe that the solution for multi-tenancy has to be dealt with at the infrastructure level. In our specific discussion, that would be the database. In this section I’d like to draw the principles for a multi-tenant data service:

• Simplicity: With a multi-tenant database, applications can be written to a single database tenant just as they are today. How tenants are allocated and shared with other tenant needs would be completely abstracted from the application.
• Efficiency: Efficiency is maintained by the fact that all data tenants can share the same hardware (memory, CPU) with other tenants.
• Dynamic scaling: Each database tenant can be distributed and scale dynamically across multiple machines, if necessary, to meet demand. This includes moving data to other machines if necessary. Scaling must be supported seamlessly without any downtime.
• Support for multiple isolation/sharing levels: With a multi-tenant database, the level of sharing/isolation is set on demand, based on the specific requirements. At that level, a user can choose a dedicated tenant, in which case the multi-tenant database would allocate tenants on a set of machines that is dedicated to that specific user. Clearly, the benefit of this is isolation, at the cost of efficiency. Another type of sharing is available to the user at the same time: Choosing to share with any individual or group under the user's control. In other words, rather then dictating either extreme (full isolation or full sharing) for all users, with a multi-tenant database we can dynamically choose the right level that best fits our purpose.

Where we are today?

Cloud providers such as Amazon, Microsoft, and Google have already realized that they can’t expect their users to deal with multi-tenancy themselves. So they've come up with data services that were designed with multi-tenancy in mind. Amazon recently introduced Amazon RDS (Relational Database as a Service) and Microsoft introduced SQL Azure. Both are an attempt to take existing databases (MySQL in the case of Amazon and SQL Server in the case of Microsoft) and add some of the characteristics that I discussed above. For example In both cases the user works with a single tenant and is completely abstracted from the underlying resource sharing optimizations. 

And yet, both solutions are still fairy limited when it comes to providing elasticity: With the Amazon solution, you're expected to scale-up (use a bigger machine) and go through a period of downtime when you switch between one degree of scaling and another. Google AppEngine has taken a different approach. Instead of relying on an existing RDBMS and wrapping it up with multi-tenancy and dynamic scaling, they took their own NoSQL alternative (BigTable) that was already built for dynamic scaling and, through a mapping layer, provided SQL semantics on top of it. As with RDS and SQL Azure, the user is completely abstracted from the way Google manages multi-tenancy in their underlying infrastructure.

Niether Amazon, Microsoft, nor Google provide WRT flexibility to the isolation level, and more importantly, all these solutions seem to be completely tied into their own cloud data center, which means that I cannot port it to my own local environment. From a throughput efficiency perspective, SQL Azure and Google seem to trade scaling for throughput by introducing an additional layer of indirection (load balancer in the case of Azure) that makes their multi-tenant solution fit only low-end applications.

At GigaSpaces we took an approach very similar to the Google approach with our new Elastic Middleware Services, where we use our existing distributed NoSQL In-Memory Data Grid and used it as a muti-tenant data service.

That is it for similarity. Here are the differences:

• You can easily download and install GigaSpaces in your local environment or use it in the cloud and get the full multi-tenancy and dynamic scaling experience. We provide an open SPI that enables you to plug in any resource pool, such as starting from a simple SSH call through a complete virtualization pool.
• You can control the level of isolation, scaling, and other SLA directly through an API.
• Our data services are designed for extreme efficiency by maximizing the utilization of memory resources. This normally translates into the ability to run at least 10x more user transactions on the same hardware, which in turn translates into huge cost savings.
• We provide the same consistent approach for other layers of the stack such as messaging and map/reduce through the same environment.

You can learn more about the Elastic Middleware solution in my previous post.

References

• SaaS Enablement of existing applications (Videocast)

• The Missing Piece in the Virtualization Stack (Part 2)

• Why Multitenancy Matters In The Cloud by Alok Misra

• Multi Tenancy - Keep It Simple, REALLY Simple by Ayende Rahien
• Multitenancy definition – Wikipedia
• Elastic Data On the Cloud: Hype or Reality 
• GigaSpaces 7.1 Release
• Multitenant Magic - Under the Covers of the Force.com Data videocast by Craig Weisseman Chief Software Architect salesforce.com
• Security and multitenancy with GigaSpaces XAP (docs)
• Security and multitenancy with GigaSpaces XAP (videocast)

The Missing Piece in the Virtualization Stack (Part 2)

In the first part of this post, I discussed how virtualization and cloud computing, as we know it today, is only a small part of the solution for today’s IT inefficiencies. While new technologies and delivery models have made it much simpler to manage the infrastructure, this is not where our core inefficiencies lie. Virtualization principles must be extended to higher levels of the application stack, to make it easier for all of us to manage, tune and integrate applications. Otherwise we will continue to spend most of our time on things that don’t provide real value to the business – infrastructure, installation, management, tuning, and integration.

In this post, as promised, I’ll show how this missing piece can be filled in practice, by something called Elastic Middleware Services. These are middleware services that can be deployed with one API call, and are completely abstracted from the applications that use them. Elastic Middleware Services are very similar to cloud-based middleware like Amazon SimpleDB, but the major difference is that they are enterprise-grade, and can be purchased and installed in your private data center, for use by your internal enterprise applications.

To better illustrate this idea, I’ll discuss a reference implementation of Elastic Middleware Services which will shortly be released as part of GigaSpaces XAP 7.1. The discussion is intended to illustrate in concrete terms how to fill the missing piece in the virtualization stack. I’ve intentionally focused on the conceptual and fundamental elements of the GigaSpaces implementation, not on specific features and benefits, to make the discussion useful even if you are not using or considering GigaSpaces products.

By the way, I recently hosted a webinar on this topic, together with Uri Cohen, GigaSpaces’ product manager. In the webinar we presented GigaSpaces’ upcoming Elastic Middleware Services. The recorded version of this presentation is available below:

Elastic Middleware Services

The idea is to provide middleware services (messaging, data, Map/Reduce, and more), like those the ones provided by cloud providers such as Amazon and Microsoft Azure, in your own local environment. These services simply extend the basic principles of virtualization, as I outlined in the previous post, to the middleware stack:

Basic virtualization principles

1. Break big physical resources into smaller logical units
2. Decouple the application from the physical resources
3. Provide an abstraction that makes all the small units look like one big unit

From a middleware perspective, this means that instead of having one big database, you have lots of small units of data services (also called partitions or shards), which are grouped together through a client side proxy that exposes all of these small units as one big unit to the client that is using them. The client proxy is also responsible for abstracting the client from the physical location of those resources. To scale, all you need to do is add more of those units. The client experiences these additional units as extra capacity added to the service.

Deploying middleware with one API call

From an API perspective, this is a much simpler approach than traditional middleware, because it introduces a higher-level abstraction to the user, as can be seen in the code snippet below:

With this new elastic middleware API, users define requirements in a language that is closer to their domain. For example, “I want to create a data service that will have 10GB of data and can grow up to 100GB of data, I need hot failover and I'm willing to share my deployment with other users of my own organization but not with users of other organizations.”

In response, the Elastic Middleware Services would automatically do the following:

• Allocate the right number of partitions, based on machine availability and the available memory in each machine, to address the capacity required.
• Automatically launch a hot backup in for each partition and ensure that it runs on separate machine than the primary partition, to address the high-availability requirement. 
• Make sure that if another organization’s application is already running on the machine, the application won’t deploy on that machine, to address the security requirement.

Fully automated through built-in SLA

As I noted in my previous post, one of the major costs of enterprise applications over a period of three years is the cost of maintenance and operations. The main component of maintenance costs is the labor cost, closely tied to the amount of manual work you have to put into your system to make sure that it meets the application’s SLA. The cost grows proportionally to the demand for scale.

With Elastic Middleware, maintenance cost can be substantially reduced through built-in SLA and automation that covers the following aspects:

• Scaling SLA – scale when there is a memory shortage, high CPU utilization, Garbage Collector hiccups, etc. Users can choose to scale automatically when any of these events occurs, and customize their own thresholds (e.g. scale when CPU utilization hits 90%). The Elastic Middleware is integrated with a cloud or virtualization framework, to enable it to automatically pull or release of machines as needed.
• Failover – when a machine fails, the Elastic Middleware does one of two things:
  • Scale down – if there are available resources to meet the current workload, the system will automatically scale down and continue to service the application.
  • Rebalance – if there are not enough resources to serve the application, the Elastic Middleware calls the cloud/virtualization pool and starts a new machine. As soon as the new machine starts, it is added to the existing pool of resources, and the application is re-balanced to take advantage of the additional capacity available through the new machine. It is important to point out that if you don’t have a dynamic pool of resources available through virtualization or cloud computing, you could still start a machine manually. Once the machine is started, adding it to the application’s resources and rebalancing would happen in exactly the same way.
• Continuous rebalancing – when new machines are added to the Elastic Middleware, it immediately detect them and re-balances the assets currently running on the new machines (if necessary), ensuring optimum utilization of all services.

Carrying out this dynamic SLA was deigned to happen while the application is running, ensuring no transaction or data loss during the process. In-flight transactions continue to be served with no noticeable hick-ups. 

Concerned of losing control?

Whenever I introduce the concept of built-in SLA and automation, I get two type of responses. One is “Wow, that’s cool!”, the other is “It sounds like I’ll lose control.” The concern about losing control is very valid, as in many cases, especially in non-cloud environments the concept of spare capacity rarely exist and therefore adding a new machine would often involve some manual intervention. In addition, when something goes wrong, other parts of the system or even other parts of the organization need to be involved, so we can’t always assume that full automation is possible.

The cruise control analogy

The approach GigaSpaces has taken is similar to the way cruise control works in our cars. You can choose to give up full control under a certain threshold, but at each point in time you can take the wheel back and resume full control. Like many of the cloud providers, we provides this type of cruise control for our services through an open API that enables management and control of every aspect of our cluster. You can query the available node, CPU, application partitions and service components. GigaSpaces Elastic Middleware was built as a layer on top of that API, which means that if the built-in functionality that comes with the Elastic Middleware doesn’t fit your needs, you can go one layer down and write your own custom behavior. For example, you can specify that in your system, scaling or failover events send an alert that will trigger a manual process for adding a new machine. In other words, you have full flexibility to choose when to turn on “cruise control” and when to take ownership.

SaaS-enabled with built-in multi-tenancy

Multi-tenancy often means that can run multiple users/customer applications on a shared resource, thereby reducing the cost of ownership per user. This is considered an area of high complexity, specifically in SaaS applications, because with today’s middleware, the application itself is responsible for mapping between users and application tenants. With Elastic Middleware Services, multi-tenancy is built into our implementation and API. Thus the burden of sharing resources is moved from the application down to the middleware. Each application works with its own dedicated middleware service (data, messaging, etc.) but that middleware service can share resources with other middleware services. In other words, instead of having one big database running on a dedicated machine and split it out between users at the application level, you can have lots of small databases spread between machines, where each application can have its own dedicated database and still have that database shared with other database instances that runs on that same machine.

This not only significantly simplifies your code, it also provides better isolation between multiple users, as well as independent life cycle management for each tenant.

There are various tradeoffs between sharing and isolation. With isolation you get better security and control of your own environment, but lower utilization and higher cost. With sharing, you can reduce cost and still achieve reasonable isolation, but not at the same level as when running on a dedicated resource. The Elastic Middleware Services make it possible to define the isolation/sharing at the application level. It does this by introducing the notion of an isolation level:

Currently we support three isolation levels:

• Dedicated – guarantees a dedicated machine allocated per instance of the application.
• Shared private – multiple instances of the application or organization share the same resources, but other departments or organizations are isolated.
• Shared public – everyone shares everything.

 

Available both on your local network and on the cloud

To try the new service you don’t need to have a private cloud or run your application on a public cloud. You don’t even need a virtualization layer. All you need is to launch a single GigaSpaces agent (Java process) per machine (normally that process will be started automatically at machine boot).

Once you’ve done this, you can start interacting with your machines and an create the desired middleware service through a simple API call. The following snippet shows how you might use this API:

Admin admin = new AdminFactory().createAdmin();

ElasticServiceManager elasticServiceManager = admin.getElasticServiceManagers().waitForAtLeastOne();

// Start a new data-grid

ProcessingUnit pu = elasticServiceManager.deploy(new ElasticDataGridDeployment("mygrid") // give it a name

.isolationLevel(IsolationLevel.DEDICATED) //isolation level

.highlyAvailable(true) // set the high availability level

.elasticity("2GB", "6GB") // set the required capacity range

.jvmSize("512MB") // configure the size per VM

.addSla(new MemorySla(70))); // define Memory SLA

That’s it! – you just got a full cluster deployed and ready to use.

What does it mean for you?

To sum up, I here are the main benefits that Elastic Middleware Services bring to each type of user:

For developers

As a developer, you can get access to the service you want just by calling an API, without worrying about installation or cluster setup. You get access to the service from your existing platform. In other words, you don’t need to run your application on GigaSpaces XAP to use any of those services. You can also pick and choose – use only the services you want, e.g. just the data, data and messaging, etc.

For the (private) data center

From a data-center perspective, you could install Elastic Middleware Services only once in your data-center. The same installation and resources would be shared amongst all users in your organization. If you happen to have virtualization in place, you can do this very easily using the GigaSpaces virtual appliances. 

Once you install the system, other users in the organization don’t need to go through the installation process – they can start consuming the middleware services just by calling an API, like they would do on the Amazon cloud when they launch a SimpleDB instance.

In a way, the Elastic Middleware Services gives the data center the power to provide higher-lever services to the business, rather than just plain infrastructure services, enabling the organization to become more agile.

For public cloud providers

Public cloud providers can be viewed as an outsourced version of the local data center, so they should experience much the same benefits as data centers. In addition, with Elastic Middlware Services, public cloud providers can offer a set of middleware services that are enterprise-ready with extremely high utilization per service. Because the Elastic Middleware Services are fully memory-based, you can use as much as 10X less machines to achieve the same throughput requirements, compared to a disk-based approach.

Ricky Ho published some interesting performance characteristics of Amazon SQS and SimpleDB in one of his recent posts:

• Network latency and throughput: 20 - 100 ms for SQS access, SimpleDB domain write throughput is 30 - 40 items/sec.
• Eventual consistency: 2 simultaneous requests to dequeue from SQS can both get the same message. SQS sometimes reports empty when there are still messages in the queue

While both SQS and SimpleDB provides a scalable data store and messaging services, their performance are at least 10-50X slower than an equivalent GigaSpaces data and messaging service. Implementing a Map/Reduce scenario with GigaSpaces is going to be significantly simpler and closer to real-time compared to Amazon Elastic Map Reduce (and that’s a topic for a separate post). Another cool feature is the ability to execute your code within the service container, which gives you an even bigger improvement in performance and latency.

The Elastic Middleware Services have much higher utilization and simpler maintenance, compared to equivalent services offered separately to the end-user, because they are provided using a shared cluster with a unified clustering model. With Elastic Middleware Services you get JPA, memcached, key/value (NoSQL), Spring, JMS, Remoting and Map/Reduce in Java or .Net, in a single deployment.

Existing GigaSpaces XAP users

Existing GigaSpaces XAP users can use the Elastic Middleware Services as part of XAP.  They’ll benefit from the simplicity provided through the high-level abstraction. In addition, the Elastic Middleware Services make it easier to plug-in the GigaSpaces components into existing application servers or development environments. They also make it significantly simpler to manage and deploy GigaSpaces XAP in various groups in the organization, because they can all share the same virtual pool of machines in the data center.

Availability

The Elastic Middleware Services will be available as part of our upcoming XAP 7.1 release, due for end of March 2010. It is currently available for private beta testing. We will also release a public beta of the Elastic Middleware Services through our upcoming 7.1 release candidate version. As always, we’ll welcome feedback and will be very happy to hear about your specific requirements. You could either post comment to this post or send an email to <pm at gigaspaces dot com>.

Why Existing Databases (RAC) are So Breakable!

One of the core assumption behind many of today’s databases is that disks are reliable. In other words, your data is “safe” if it is stored on a disk, and indeed most database solutions rely heavily on that assumption. Is it a valid assumption?

Disks are not reliable!

Recent analysis of disk failure patterns (see references below) break the disk reliability assumption completely. Interestingly enough, even though these studies were published in 2007, most people I speak to aren’t aware of them at all. In fact, I still get comments from people who are basing their reliability solution on disk-based persistency, insisting on the argument that whether or not you are reliable amounts to whether or not your data is synched to a disk.

Below are some of the points I gathered from the research, which should surprise most of us:

• Actual disk failure/year is 3% (vs. estimates of 0.5 - 0.9%) – this is a 600% difference on reported vs. actual disk failure. The main reason is that manufacturers can’t tests their disks for years, so their tests are an extrapolation of statistical behavior which proves to be completely inaccurate.
• There is NO correlation between failure rate and disk type – whether it is SCSI, SATA, or fiber channel. Most data centers are based on the assumption that investing in high-end disks and storage devices will increase their reliability – well, it turns out that high-end disks exhibit more or less the same failure patterns as regular disks! John Mitchell had an interesting comment on this matter during our Qcon talk, when someone pointed to RAID disks as their solution for reliability. John said that since RAID is based on an exact H/W replica that lives in the same box, there is a very high likelihood that if a particular disk fails, its replica will fail in the same way. This is because they all have the exact same model, handle the exact same load and share the same power/temperature. 
• There is NO correlation between high disk temperature and failure rates – I must admit that this was a big surprise to me, but it turned out that there is no correlation between disk temperature and disk failure. In fact, most failures happen when disks were at low or average temperature.

Why are existing database clustering solutions so breakable?

Most existing database clustering solutions rely on a shared disk storage to maintain their cluster state, as can be seen in the diagram below.

 

Typical database clustering architecture based on shared storage

The core, implicit assumption behind Oracle RAC, IBM DB2, and many other database clustering solutions, is that failure can be avoided by purchasing high-end disk storage and using expensive hardware (fiber optics, etc). As can be seen from the research I mentioned earlier, this core assumption doesn’t correlate with the failure statistics. Hence I argue that the database clustering model is inherently breakable.

Failure is inevitable – cope with it

Jason McHugh, a senior engineer at Amazon who works on S3, gave a very interesting talk at the Qcon conference (Amazon S3: Architecting For Resieliency In The Face Of Failures), where he outlined the service failure handling architecture of Amazon’s Simple Storage. Jason’s main point was that rather than trying to prevent failure from happening, you should assume that failure is inevitable and instead, design your system to cope with it.

Failure can happen in various shapes and forms:

• Expect drives to fail
• Expect network connections to fail (independent of the redundancy in networking)
• Expect a single machine to go out

Human error example: Failure to use caution while pushing a rack of servers

How to design your application for resilience in the face of failure

Jason outlined some of the lessons from his experience at Amazon:

• Expect and tolerate failures – build your infrastructure to cope with catastrophic failure scenarios: "Distributed systems will experience failures, and we should design systems which tolerate rather than avoid failures. It ends up being less important to engineer components for low failure rates (MTBF) than it is to engineer components with rapid recovery rates (MTTR)." (from Architecture Summit 2008 - Write up)
• Code for large-scale failures – design your application to cope with the fact that services can come and go.
• Expect and handle data and message corruption – don’t rely only on TCP to ensure the consistency of your data. Add validation as part of your application code to make sure that messages are not corrupted.
• Code for elasticity – load/demand may grow unexpectedly. Elasticity is essential to dealing with unexpected demand without bringing the system to its knees. It is also important that when you exceed your capacity, the system won’t crash but instead report denial of service.
• Monitor, extrapolate and react – monitoring at the right level enables you to detect failures before they happen and take the appropriate corrective actions.
• Code for frequent single machine failures – rack failure can cause an immediate disappearance of multiple machines at the same time.
• Game days – simulate failures and learn how to continuously improve your operational procedure for handling failures and architecture.

Memory can be more reliable then disk

Many people assumes that memory is an unreliable data storage.  That assumption holds true if your data “lives” on a single machine; in this case if the machine fails or crashes your application crashes. But what if you distribute the data across a cluster of nodes and maintain more than one copy of the data over the network? In this case, if a node crashes the data is not gone; it lives elsewhere and can be continuously served from one of its replicas.

Under these conditions, I can argue that under certain conditions, memory can be more reliable than disk. One of the reason is the hardware itself – unlike disk drives that rely on mechanical moving parts, RAM is just a silicon chip. As such the chances for failure can be significantly smaller. Unfortunately, I don’t have the data to back up this argument, but I think it’s very likely that this is the case. In an article published on InfoQ titled RAM is the new disk, Tim Bray brings an interesting observation on this matter:

Memory is the new disk! With disk speeds growing very slowly and memory chip capacities growing exponentially, in-memory software architectures offer the prospect of orders-of-magnitude improvements in the performance of all kinds of data-intensive applications. Small (1U, 2U) rack-mounted servers with a terabyte or more or memory will be available soon, and will change how we think about the balance between memory and disk in server architectures. Disk will become the new tape, and will be used in the same way, as a sequential storage medium (streaming from disk is reasonably fast) rather than as a random-access medium (very slow). Tons of opportunities there to develop new products that can offer 10x-100x performance improvements over the existing ones.

As a matter of fact we (GigaSpaces/Cisco) are working these very days on benchmarking  Cisco UCS, where we are evaluating the possibility to manage x100GB and potentially even Terabytes of data in a single node. The results look extremely promising, so the future is not that far ahead.

References

• KHB: Real-world disk failure rates: surprises, surprises, and more surprises by Valerie Aurora (Henson)
• Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? By Bianca Schroeder and Garth A. Gibson
• The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM By Department of Computer Science Stanford University
  
• Amazon S3: Architecting For Resieliency In The Face Of Failures By Jason McHugh
• Failure Trends in a Large Disk Drive Population - Google, Inc., February 2007
• InfoQ Article - RAM is the new disk...
• Are Cloud Based Memory Architectures the Next Big Thing?, highscalability.com

The Interactive Cloud (Part II - Adding Operational Awareness to your Application)

In part I of of this topic, I covered the general concept and motivation behind the "Interactive Cloud". I described how the interactive cloud enables applications to be managed in a similar way to the way we run our business. In this post, I discuss further details of what it means to add operational awareness to the application. To do that, I refer to our specific experience in GigaSpaces. Towards the end of the post, I point out specific code snippets as an illustration of that model.

Background

GigaSpaces 7.0 Application Cluster Management API is an API for managing application clusters, and for giving applications operational awareness. The reason why we came up with this API in the first place, is quite interesting. When we started GigaSpaces, a typical deployment was primary-backup with multiple clients connected to this pair. Sooner then we anticipated, we found ourselves supporting cluster deployments of 10s and 100s of nodes per application. We are now at a point where customers are starting to use GigaSpaces XAP as a service shared between multiple applications, which in itself increases the deployment size quite significantly.

In this type of environment, troubleshooting a production system can be a real nightmare for everyone involved. We started to add all sorts of metrics and logging information into our system but it hardly improved things. The process of finding the right information at the level we needed was still a heavy-duty task.

When we looked back and tried to analyze how we learn about the application’s behavior, we saw that quite often we identified the time when things started to go wrong, and then, using the timestamps on the different operations, ran a complex correlation process to see what happened on each machine during that time. In many cases when we hit the first area of trouble, we were still missing information, because we were running without fine-grained logging, for performance reasons. So then we would turn up the log level and wait for the next event, and so forth.

We realized that just adding more probes and log levels and using some sort of CEP (Complex Event Processing) to parse all the garbage we were going to generate, on a continuous basis, was not going to be good enough. Even worse, if the reporting system got so complex, we would eventually end up going back to the old logs because the reporting system would inevitably lose some of our events :)

We figured that we could not afford to continue down that path – the cost, both on our end and on the customer’s end, was way too high. We needed to try a totally new approach.

The Ideal Solution – Getting Help from the Applications themselves

When we analyzed what would be the ideal process, we came to the realization that the application should be responsible for the monitoring. When the application starts to behave abnormally (for example, CPU spikes, GC spikes), or experiences network failure, it should do two things:

1. Take corrective action – the application should figure out if the problem is that resources have been exhausted. If that is the case, it should try to rebalance itself to reduce the load automatically.
2. Provide information to help analyze the root cause of the event that occurred – the application should point out the machines that are suspects and ask those server instances to increase their logging level and print a snapshot of thread dump. This provides only the information that is actually needed to perform root cause analysis, and at the right level of granularity.

It wasn’t surprising that some of the root cause analysis indicated that one of the biggest contributors to failures, is human error or misconfiguration. That led us to the realization that the way to reduce those factors would be through complete automation of manual deployment and maintenance processes.

Another thing we realized during that process, is that when things went wrong, quite often the operation guys became fairly useless at a fairly early stage of the process, and the application guys had to be called in to understand what happened. Not surprisingly, the application guys didn’t even know how to get the required information or didn’t have permission to get that information, and had to rely on someone else to do that for them – which made the entire process really long and complex.

All of this led us to the realization that we needed to create an application cluster management and monitoring tool that could be used by both the application developers and the operation guys. The application guys could use that API as part of their development cycle, and the operation guys could use it to receive the right level of information into their management and control dashboards.

In the following section, I provide a short sneak peek into this API, and how it is structured.

XAP’s New Application Cluster Management and Monitoring API (a.k.a. the Administration API)

There are basically two ways to design  the API for this challenge: define the interaction model and drive the API from there, or define the domain model and drive the interaction model from the domain model. We chose the latter.

1. Defining the Application Management Domain Model

The first step was to define the domain model of our application management. The diagram below maps out the “nervous system” of an application running on XAP – it shows the core layers of the application management domain model.

 

2. Defining the Interaction Model – the API

Once we mapped the nerves and sensors of our application, we could start to wire them. This enabled us not only to monitor the state of the application components, but also to interact with them when we needed to take corrective actions (see snippet below).

3. Making Operational Awareness an Integral Part of the Development Framework

If we want the application to take more responsibility for its operational behavior, we can’t just point it to an external management and monitoring tool that doesn’t speak the “developer language”.  We need to make the proactive and reporting behavior an integral part of the development framework. This is illustrated in the examples below.

Application Code that uses the Cluster Management API - a Few Examples

Below are a few examples that show how applications interact with the new Administration API, using a groovy-based API. These snippets were written in Groovy but they can be be called in Java as well.

Example 1 – Getting Statistics

This example uses Groovy closures to get callbacks when a new container or machine joins the network, and when a specific application event happens.

Example 2 – Calculating Average Load on a Web Container

This code snippet shows how to get an aggregated result of the total number of requests/sec from all the web containers in the cluster.

Example 3 – Scaling Up:

The code snippet below shows the process of scaling out, when the number of requests exceeds the maximum number of requests allowed per instance.

Note that the call to incrementInstance() generates an entire match-making process behind the scene. What happens is that a new web container is spawned, but the provisioning of this new container is based on the SLA that was defined at deployment time. In our case, it is deployed only on machines that don’t already run a web container and have spare CPU and memory capacity.

Initial Success Stories

Since we came up with the API, we started to see its use spreading in areas we didn’t even think of. Here are a few examples:

• Helping customers reproduce problem scenarios – now that we can script almost every part of the application, we can easily script even very complex scenarios, where the application starts to execute a few things and then crashes only when a certain operation happens, etc. The fact that we can script these scenarios makes it easier to create a reproduction scenario, and from there, the road to spotting the problem and fixing it becomes significantly shorter (see detailed reference here).
• Third part integration – one of our partners created a fairly extensible Eclipse plug-in. The reason he was able to do that pretty much by himself, was because the API gave him everything he needed in order to control the application development and testing lifecycle (see more details in Jeroen Remmerswaal post here).
• Writing a complex SLA – we were able to easily write complex SLAs, for example, automating deployment of applications across multiple data centers and making sure that primaries and backups never run on the same data center. All that without any human intervention.

What’s Coming Next?

One of the immediate things we are going to enhance, is to provide more built-in SLAs, and a set of utilities that perform deployment and implementation of mainstream scenarios, such as auto-scaling of web applications and auto re-balancing of data grid deployments. This is going to serve as a cornerstone of our new cloud-enabled middleware initiative, where all the classic, enterprise-grade middleware components become available as a service. In fact, some of our customers are already using our In-Memory Data Grid as a shared service between multiple applications within the organization. The idea is to make that type of deployment simpler and intuitive.

Take Part in this New Initiative

I can go on with more of my thoughts on what should be done, but I’d rather stop here and ask all of you for your feedback. If you’ve faced similar challenges, how do you see this solution? What would you want to see in this new package?

For those who want to try out this new API, it is worth noting that it is actually provided for free as part of our Community Edition, so anyone can simply download it and try out the examples from this post.

The Interactive Cloud (Part I)

When we think about cloud computing, the first thing that comes to mind is on-demand computing. Indeed, cloud computing and virtualization providers made it easy to get a machine by just providing a credit card and making an API call. However, that being said, what happens after you get that pool of machines? How do you deploy the application into these machines? How do you know how the application behaves? How can you react when something breaks or when the load on the system gets too high?

In the human world, many organizations face pretty much the same challenges. It is fairly easy to get a good infrastructure for business – office space, electricity, telephones, computers – but as a manager, it’s really difficult to place the right people in those offices, give them sufficient work but not so much that they’re totally overloaded, track what they’re doing, and react properly when something goes wrong. A recent and very successful methodology for dealing with this challenge, which is quickly gaining popularity, is called Agile, Scrum,  or Lean. The common ground of this family of methodologies is that unlike the centralized management approach, we work through short iterations and continuously correct the course of our actions based on the actual progress. We rely heavily on the individuals to take responsibility for their actions and be part of the decision-making process. We assume that those individuals will take their own corrective actions for things that fall under their direct responsibility. The program manager is only responsible to make sure that the course (and budget) is kept on the right target and shift resources or allocate new ones if needed.

Today’s Operations and Applications – Just Like Centralized Management

Chances are that you are aware of the Agile methodology and recognize that it solves many of the huge difficulties of traditional, centralized management. But at the same time, if you think about it, you are still running your applications and IT operations according to the defunct, centralized management paradigm. We’ve invested years in creating an artificial separation between what is known as “operations” and “applications”. In this way of thinking, “applications” is responsible for maintaining a certain business logic and “operations” is responsible for monitoring and maintaining the application. Operations management is responsible for monitoring the CPU, memory and every individual aspect of the network configuration, application configuration, and consumption of resources. The application is not really aware of any of this, and usually you rely on the operations guys to take corrective action, or at least give you a call and let you know, when something out of the ordinary happens.

This seems natural because it’s a pattern we’ve all become accustomed to. But just imagine managing a real life project (with people) using this same approach. It would take the centralized management approach to frightening extremes. Here’s how it would work – there is the project manager (operations) and the employees (application), with absolutely no direct communication between them. Each employee has a set of sensors attached to her, which report on health conditions (heartbeat, blood pressure, and so on), and another set of sensors that report how quickly she is working. The project manager views all this data in a nice dashboard with nice graphs and all sorts of reporting schemes. Whenever there is an alert – for example, one of the employees is running a fever, or is working very slowly for some reason, the project manager troubleshoots the issue and sees how to solve it and whether it affects the project.

That’s ridiculous, isn’t it!?

In real life, especially when working with Agile, employees are responsible for their output. If they don’t feel well, or are working slower than usual, only the individual employee can really know what the problem is – or whether there is a problem at all. They take a rest or figure out what else they can do about it. They only report back to the project manager if they really can’t deal with the issue themselves and it’s starting to affect their ability to meet the project goals. The manager doesn’t need to constantly monitor their health or throughput – the only thing worth monitoring is if they can or can’t do their job. The reason is less of interest. More importantly, the manager relies on the employees to report when such an event happens, and to take corrective action themselves.

If that’s so obvious, why can’t we apply the same same common sense to the way we run our software? Why can’t we make our applications aware of their condition and able to take some responsibility for themselves? This is important in any computing environment, but I would argue that it is especially important on a cloud or in any virtualized environment, in which the operations/application paradigm totally breaks down.

The Solution: Bringing Operational Awareness to your Applications

Like a real-life employee, the application needs to be responsible for its own health. It needs to report only on general status and any shortage of resources. The operation guys don’t need to continuously monitor the health condition of each individual application component. They should rely on the applications themselves to send the right warning signals before something breaks, and suggest corrective actions in a language that the operation guys can understand. For example: “I need X amount of additional resources ” or “I can release Y amount of resources after I've done my duty.” Deciding which specific task needs to be assigned to each of the available resources should be the application’s responsibility – in any case, it probably knows best how to utilize these resources. 

To make this a reality, you need to add operational awareness to your applications. By operational awareness I mean that the application needs to be written with the assumption that it can be asked to be moved at any point in time, and the application needs to take responsibility for events that can be solved at the application level, and take corrective actions to meet its SLA. The manager should only be notified in cases where the application has exhausted its physical resources and there are no other available resources. For example, if one of the instances of the application runs out of memory or needs more CPU power, and there are spare resources that are not utilized by other resources within the application, it is the application’s responsibility to balance its resources within its own realm. But even if there are no available resources, the manager doesn’t need to bother with the fine-grained details of how to balance the individual application resources. Balancing these resources should be kept within the scope of the application itself. The manager should only bother to allocate the pool of resources between the various applications.

Learning from Amazon EC2

A working example of this approach is the Amazon EC2 API. In my view, one of the revolutionary concepts that made EC2  so successful, was the realization that rather then creating high level tools for managing the data center, Amazon chose to provide an API and let the application developers figure out how to run their application. The API opened up a whole ecosystem of third party solutions, and a whole new market. What’s interesting is that in EC2, when I want to auto-scale my application, I don’t call the operations guys to do that. There is a very slim chance that someone who doesn’t understand my application would be able to write such a policy without breaking the application. Most of the applications that run on EC2 just call the EC2 API when they want to spawn a new machine or manage their high availability, or auto-scaling of their application. The nice thing about the API is that it doesn’t prevent Amazon and others coming up with more high-level tools that are much more useful for the operation guys at a later stage. The main difference is that the application takes more responsibility for ensuring that it meets its SLA, and the operations guys get the type of information that makes sense to them, and more importantly, something that they can easily act upon and take the appropriate corrective actions.

The Need for an Application Cluster Management API

Amazon provides a comprehensive set of APIs that work at the Virtual Machine level. Back to our real-life project analogy, the Amazon tool provides us with the real-estate  - the office space, the network, the computers. What we don’t get is the access to the individuals that do the actual work. For this reason, keeping the API at the virtual machine level is often not granular enough, as it doesn’t enable us to interact with the individual application components, manage and automate the application’s deployment lifecycle, define its dependencies and distribution model. There are other frameworks such as Capistrano that became popular, mostly in managing Ruby deployments. This type of tool is becoming popular in other cloud deployment settings as well. Capistrano can address the application deployment and basic life cycle management, and provide a good foundation for automation of those processes.

While the combination of the two is a good starting point, it is lacking the notion of application cluster. In an application cluster, you want to have more explicit semantic support for managing deployment topologies (partitioning, replicated). You need built-in support for defining backup and primary, and you need to be aware of the data affinity requirement of certain elements in the architecture. Without all this, enabling the application to re-balance its resources when a new machine joins or leaves the network, is close to impossible.

The Interactive Cloud

The combination of image-level provisioning and an application cluster management API, leads to a new form of cloud deployment experience that I refer to as the interactive cloud.

The interactive cloud operates fairly closely to the way we manage human-led projects. The image API is equivalent to the physical resources that are critical to run our project, e.g. the office, the computer, and the network. Another part of the project is managing the individuals that run the project. Learning the right way to manage our teams, from the lessons of Agile project management, is through a close iterative process. This is where the Application Cluster Administration API comes into play. This API gives the application developer the ability to control and automate application deployment, and continuously balance the way the application runs, to ensure that it meets its required SLA. It also provides a callback mechanism when you run out of resources and when some of the idle resources can be released. You can also easily generate an aggregated view for operations (without going down to the level of what each individual component did at each given point in time).

With this combination, you can spawn new applications, automate their deployment and lifecycle, start new machines as needed and keep a close watch on your application behavior after it has been deployed. When a new machine joins the cluster, the application automatically rebalances its resources to take advantage of that new machine, and when machines fail or are removed from the cluster, the application rebalances itself on the smaller resource pool. If something breaks, the application automatically scales down, allowing it to continue working despite the failure, and in the background allocate an alternate resource to fit in instead of the new one.

IMO, a cloud/virtualized environment forces us to go in this direction. This is an area where the old way of separating “applications” from “operations” is just not going to cut it.

Final Words

There is nothing really new about the solution I propose here, and the concept behind it. Quite the contrary. The interactive cloud allows you to apply proven methodologies that work well for many critical projects in the real world, and apply them to the way you run your applications. Most people who run their application in the cloud have already started to build and manage their applications in this way, using tools like the Amazon EC2 API and Capistrano. However, this was mostly done as an afterthought and was mainly driven by intuition and common sense. My attempt in this post is to back up these initial efforts with a clear methodology.

In my next post (Adding operational awareness to your application), I will use the Administration and Monitoring API, new in GigaSpaces XAP 7.0, as an example for such a framework. It is worth noting at this point that this API is actually provided for free as part of our Community Edition, so anyone can simply download it and try out the examples I will provide in the next post. I’m looking forward to showing that this vision of the interactive cloud is really very tangible and is already at everyone’s fingertips.