Nati Shalom's Blog

How a Top Financial Firm Uses XAP for Dynamic Scaling and How 7.1 Makes It Simpler than Ever

GigaSpaces XAP 7.1 was officially announced last week. One of the highlight of the release is the new elastic middleware capabilities that makes scaling and fail-over significantly simpler. 

Today  (April 21, 2010) i will present a case study that illustrates how the Elastic Middleware and Admin API that comes with 7.1 helped one of the top wall street firms to manage a fairly complex planned and unplanned downtime scenario of one of their real-time web application and through that reduce their  potential downtime cost associated with this scenarios. (According to IDC, the total cost of downtime in 2009 is estimated at $400 billion, an average of $8,000 per server per year)

During the session i will also un-evil  one of the recent large memory benchmarks that we conducted with new Cisco UCS machine and how it made it possible to manage large memory deployments with x100GB per JVM instance.

If time permits I'll show sneak preview of our upcoming scalability  blue-print demos for real-time web application.  The demo shows a combination of real-time web application (Stock viewer in this specific case) connected to a real-time analytics data service that collects real time feeds and measures the historical trend analysis of selected stocks. The demo will illustrate how the use of the elastic middleware can enable us to  accumulate few days worth of data in-memory and increase our in-memory capacity on-demand while the system is running.

 

As always this webinar could be a great opportunity to hear your feedback and answer your questions on those and other topics. The registration details are provided below.

• Register for Early Session  - 9:30AM CEST / 5:30PM Australia (EST)
• Register for Late Session - 9:30AM PDT / 5:30PM BST
  

The Missing Piece in the Virtualization Stack (Part 2)

In the first part of this post, I discussed how virtualization and cloud computing, as we know it today, is only a small part of the solution for today’s IT inefficiencies. While new technologies and delivery models have made it much simpler to manage the infrastructure, this is not where our core inefficiencies lie. Virtualization principles must be extended to higher levels of the application stack, to make it easier for all of us to manage, tune and integrate applications. Otherwise we will continue to spend most of our time on things that don’t provide real value to the business – infrastructure, installation, management, tuning, and integration.

In this post, as promised, I’ll show how this missing piece can be filled in practice, by something called Elastic Middleware Services. These are middleware services that can be deployed with one API call, and are completely abstracted from the applications that use them. Elastic Middleware Services are very similar to cloud-based middleware like Amazon SimpleDB, but the major difference is that they are enterprise-grade, and can be purchased and installed in your private data center, for use by your internal enterprise applications.

To better illustrate this idea, I’ll discuss a reference implementation of Elastic Middleware Services which will shortly be released as part of GigaSpaces XAP 7.1. The discussion is intended to illustrate in concrete terms how to fill the missing piece in the virtualization stack. I’ve intentionally focused on the conceptual and fundamental elements of the GigaSpaces implementation, not on specific features and benefits, to make the discussion useful even if you are not using or considering GigaSpaces products.

By the way, I recently hosted a webinar on this topic, together with Uri Cohen, GigaSpaces’ product manager. In the webinar we presented GigaSpaces’ upcoming Elastic Middleware Services. The recorded version of this presentation is available below:

Elastic Middleware Services

The idea is to provide middleware services (messaging, data, Map/Reduce, and more), like those the ones provided by cloud providers such as Amazon and Microsoft Azure, in your own local environment. These services simply extend the basic principles of virtualization, as I outlined in the previous post, to the middleware stack:

Basic virtualization principles

1. Break big physical resources into smaller logical units
2. Decouple the application from the physical resources
3. Provide an abstraction that makes all the small units look like one big unit

From a middleware perspective, this means that instead of having one big database, you have lots of small units of data services (also called partitions or shards), which are grouped together through a client side proxy that exposes all of these small units as one big unit to the client that is using them. The client proxy is also responsible for abstracting the client from the physical location of those resources. To scale, all you need to do is add more of those units. The client experiences these additional units as extra capacity added to the service.

Deploying middleware with one API call

From an API perspective, this is a much simpler approach than traditional middleware, because it introduces a higher-level abstraction to the user, as can be seen in the code snippet below:

With this new elastic middleware API, users define requirements in a language that is closer to their domain. For example, “I want to create a data service that will have 10GB of data and can grow up to 100GB of data, I need hot failover and I'm willing to share my deployment with other users of my own organization but not with users of other organizations.”

In response, the Elastic Middleware Services would automatically do the following:

• Allocate the right number of partitions, based on machine availability and the available memory in each machine, to address the capacity required.
• Automatically launch a hot backup in for each partition and ensure that it runs on separate machine than the primary partition, to address the high-availability requirement. 
• Make sure that if another organization’s application is already running on the machine, the application won’t deploy on that machine, to address the security requirement.

Fully automated through built-in SLA

As I noted in my previous post, one of the major costs of enterprise applications over a period of three years is the cost of maintenance and operations. The main component of maintenance costs is the labor cost, closely tied to the amount of manual work you have to put into your system to make sure that it meets the application’s SLA. The cost grows proportionally to the demand for scale.

With Elastic Middleware, maintenance cost can be substantially reduced through built-in SLA and automation that covers the following aspects:

• Scaling SLA – scale when there is a memory shortage, high CPU utilization, Garbage Collector hiccups, etc. Users can choose to scale automatically when any of these events occurs, and customize their own thresholds (e.g. scale when CPU utilization hits 90%). The Elastic Middleware is integrated with a cloud or virtualization framework, to enable it to automatically pull or release of machines as needed.
• Failover – when a machine fails, the Elastic Middleware does one of two things:
  • Scale down – if there are available resources to meet the current workload, the system will automatically scale down and continue to service the application.
  • Rebalance – if there are not enough resources to serve the application, the Elastic Middleware calls the cloud/virtualization pool and starts a new machine. As soon as the new machine starts, it is added to the existing pool of resources, and the application is re-balanced to take advantage of the additional capacity available through the new machine. It is important to point out that if you don’t have a dynamic pool of resources available through virtualization or cloud computing, you could still start a machine manually. Once the machine is started, adding it to the application’s resources and rebalancing would happen in exactly the same way.
• Continuous rebalancing – when new machines are added to the Elastic Middleware, it immediately detect them and re-balances the assets currently running on the new machines (if necessary), ensuring optimum utilization of all services.

Carrying out this dynamic SLA was deigned to happen while the application is running, ensuring no transaction or data loss during the process. In-flight transactions continue to be served with no noticeable hick-ups. 

Concerned of losing control?

Whenever I introduce the concept of built-in SLA and automation, I get two type of responses. One is “Wow, that’s cool!”, the other is “It sounds like I’ll lose control.” The concern about losing control is very valid, as in many cases, especially in non-cloud environments the concept of spare capacity rarely exist and therefore adding a new machine would often involve some manual intervention. In addition, when something goes wrong, other parts of the system or even other parts of the organization need to be involved, so we can’t always assume that full automation is possible.

The cruise control analogy

The approach GigaSpaces has taken is similar to the way cruise control works in our cars. You can choose to give up full control under a certain threshold, but at each point in time you can take the wheel back and resume full control. Like many of the cloud providers, we provides this type of cruise control for our services through an open API that enables management and control of every aspect of our cluster. You can query the available node, CPU, application partitions and service components. GigaSpaces Elastic Middleware was built as a layer on top of that API, which means that if the built-in functionality that comes with the Elastic Middleware doesn’t fit your needs, you can go one layer down and write your own custom behavior. For example, you can specify that in your system, scaling or failover events send an alert that will trigger a manual process for adding a new machine. In other words, you have full flexibility to choose when to turn on “cruise control” and when to take ownership.

SaaS-enabled with built-in multi-tenancy

Multi-tenancy often means that can run multiple users/customer applications on a shared resource, thereby reducing the cost of ownership per user. This is considered an area of high complexity, specifically in SaaS applications, because with today’s middleware, the application itself is responsible for mapping between users and application tenants. With Elastic Middleware Services, multi-tenancy is built into our implementation and API. Thus the burden of sharing resources is moved from the application down to the middleware. Each application works with its own dedicated middleware service (data, messaging, etc.) but that middleware service can share resources with other middleware services. In other words, instead of having one big database running on a dedicated machine and split it out between users at the application level, you can have lots of small databases spread between machines, where each application can have its own dedicated database and still have that database shared with other database instances that runs on that same machine.

This not only significantly simplifies your code, it also provides better isolation between multiple users, as well as independent life cycle management for each tenant.

There are various tradeoffs between sharing and isolation. With isolation you get better security and control of your own environment, but lower utilization and higher cost. With sharing, you can reduce cost and still achieve reasonable isolation, but not at the same level as when running on a dedicated resource. The Elastic Middleware Services make it possible to define the isolation/sharing at the application level. It does this by introducing the notion of an isolation level:

Currently we support three isolation levels:

• Dedicated – guarantees a dedicated machine allocated per instance of the application.
• Shared private – multiple instances of the application or organization share the same resources, but other departments or organizations are isolated.
• Shared public – everyone shares everything.

 

Available both on your local network and on the cloud

To try the new service you don’t need to have a private cloud or run your application on a public cloud. You don’t even need a virtualization layer. All you need is to launch a single GigaSpaces agent (Java process) per machine (normally that process will be started automatically at machine boot).

Once you’ve done this, you can start interacting with your machines and an create the desired middleware service through a simple API call. The following snippet shows how you might use this API:

Admin admin = new AdminFactory().createAdmin();

ElasticServiceManager elasticServiceManager = admin.getElasticServiceManagers().waitForAtLeastOne();

// Start a new data-grid

ProcessingUnit pu = elasticServiceManager.deploy(new ElasticDataGridDeployment("mygrid") // give it a name

.isolationLevel(IsolationLevel.DEDICATED) //isolation level

.highlyAvailable(true) // set the high availability level

.elasticity("2GB", "6GB") // set the required capacity range

.jvmSize("512MB") // configure the size per VM

.addSla(new MemorySla(70))); // define Memory SLA

That’s it! – you just got a full cluster deployed and ready to use.

What does it mean for you?

To sum up, I here are the main benefits that Elastic Middleware Services bring to each type of user:

For developers

As a developer, you can get access to the service you want just by calling an API, without worrying about installation or cluster setup. You get access to the service from your existing platform. In other words, you don’t need to run your application on GigaSpaces XAP to use any of those services. You can also pick and choose – use only the services you want, e.g. just the data, data and messaging, etc.

For the (private) data center

From a data-center perspective, you could install Elastic Middleware Services only once in your data-center. The same installation and resources would be shared amongst all users in your organization. If you happen to have virtualization in place, you can do this very easily using the GigaSpaces virtual appliances. 

Once you install the system, other users in the organization don’t need to go through the installation process – they can start consuming the middleware services just by calling an API, like they would do on the Amazon cloud when they launch a SimpleDB instance.

In a way, the Elastic Middleware Services gives the data center the power to provide higher-lever services to the business, rather than just plain infrastructure services, enabling the organization to become more agile.

For public cloud providers

Public cloud providers can be viewed as an outsourced version of the local data center, so they should experience much the same benefits as data centers. In addition, with Elastic Middlware Services, public cloud providers can offer a set of middleware services that are enterprise-ready with extremely high utilization per service. Because the Elastic Middleware Services are fully memory-based, you can use as much as 10X less machines to achieve the same throughput requirements, compared to a disk-based approach.

Ricky Ho published some interesting performance characteristics of Amazon SQS and SimpleDB in one of his recent posts:

• Network latency and throughput: 20 - 100 ms for SQS access, SimpleDB domain write throughput is 30 - 40 items/sec.
• Eventual consistency: 2 simultaneous requests to dequeue from SQS can both get the same message. SQS sometimes reports empty when there are still messages in the queue

While both SQS and SimpleDB provides a scalable data store and messaging services, their performance are at least 10-50X slower than an equivalent GigaSpaces data and messaging service. Implementing a Map/Reduce scenario with GigaSpaces is going to be significantly simpler and closer to real-time compared to Amazon Elastic Map Reduce (and that’s a topic for a separate post). Another cool feature is the ability to execute your code within the service container, which gives you an even bigger improvement in performance and latency.

The Elastic Middleware Services have much higher utilization and simpler maintenance, compared to equivalent services offered separately to the end-user, because they are provided using a shared cluster with a unified clustering model. With Elastic Middleware Services you get JPA, memcached, key/value (NoSQL), Spring, JMS, Remoting and Map/Reduce in Java or .Net, in a single deployment.

Existing GigaSpaces XAP users

Existing GigaSpaces XAP users can use the Elastic Middleware Services as part of XAP.  They’ll benefit from the simplicity provided through the high-level abstraction. In addition, the Elastic Middleware Services make it easier to plug-in the GigaSpaces components into existing application servers or development environments. They also make it significantly simpler to manage and deploy GigaSpaces XAP in various groups in the organization, because they can all share the same virtual pool of machines in the data center.

Availability

The Elastic Middleware Services will be available as part of our upcoming XAP 7.1 release, due for end of March 2010. It is currently available for private beta testing. We will also release a public beta of the Elastic Middleware Services through our upcoming 7.1 release candidate version. As always, we’ll welcome feedback and will be very happy to hear about your specific requirements. You could either post comment to this post or send an email to <pm at gigaspaces dot com>.

The Missing Piece in the Virtualization Stack (Part 1)

This and the next post will discuss how virtualization and cloud computing, as we know it today, is only a small part of the solution for today’s IT inefficiencies. While new technologies and delivery models have made it much simpler to manage the infrastructure, this is not where our core inefficiencies lie. Virtualization principles must be extended to higher levels of the application stack, to make it easier for all of us to manage, tune and integrate applications. Otherwise we will continue to spend most of our time on things that don’t provide real value to the business.

What do we really spend our time on?

If you’ve been in the application development space for a while, I'm sure that you are all familiar with the current application development cycle. The diagram below shows a typical application development cycle. As you can see, we spend a large part of our time on things that don’t provide real value to our business.

Typical application development lifecycle

The continuous demand for scale and scalability has made things even worse – many of us are forced to repeat this cycle over and over again every time we are faced with new scaling requirements:

The promise of virtualization/cloud

Virtualization and cloud computing aim to solve a large part of the overhead involved in setting up the infrastructure (buying new hardware, setting it up, installing it, etc). Indeed, we can now start a new machine just by calling an API, we can lease a machine or even completely outsource our entire infrastructure to a public hosting provider.

Does this solve all of our problems?

As I outlined in the diagram above, setting up the infrastructure is only part of the challenge in the development of new business application. If you measure the complexity/effort required, plugging an application into the infrastructure isn’t necessarily the biggest challenge. Most of us spend most of our time maintaining our code, plumbing it to other services within our organizations and continuously maintaining and tuning it. In recent years, with the growth of data volumes on the one hand and the demand for better efficiency on the other hand, I found that most of the time (and cost!) is spent on dealing with these two contradicting requirements: Each demand for additional scaling forces us to go through a complete cycle of tuning, design and in some cases, through a complete product selection phase to meet the demand.

Last week alone, I found myself spending a good amount of time in discussion with a large telco ISV that built its solution through a combination of storage devices, databases, and so on. In the telco world many of these services face both an increase in the size of the data (per user) and an increase in the number of users. Imagine the increase in the size of pictures that you’re able to send through your phone. It started with few KB, is now up to 100KB and will soon get up to Megabytes of data per message, as the camera resolution grows. Multiply it by the number of users and messages per second and you get a classic scaling challenge. In this telco ISV’s specific case, it is fairly easy to partition the problem based on users (personally, I believe that this is only a temporary assumption, as I'm sure that with the likes of Twitter this assumption will no longer hold true). Now, they could have gone through the traditional way of scaling which is to duplicate their system several times, each unit dealing with smaller amount of users. Now that sounds easy, so why they are still reluctant to do that? The answer is fairly easy – cost. This is potentially an easy solution but fairly inefficient. Taking into account that there is only so much a customer would be willing to pay, that cost will come out of their pocket and effect their profit margin. It may even cost them their business, as at this point they might be beaten by a competitor who comes up with a more efficient solution.

The elephant in the room

Now here is the elephant in the room – would virtualization or cloud solve their problem? It might be a solution – but for only a small part of their challenge. And that’s my point. We spent a good deal of the past year talking about cloud and virtualization as the solution for all of our inefficiency problems, but we forgot that they cover only a small part – in some cases even a fairly insignificant part – of our challenge. Our main real-life challenge is not how to make our infrastructure more efficient but how to make our business more efficient!

To illustrate this gap, I like to use the three questions below:

Assuming that with cloud and virtualization you can easily create a new machine by a call of an API…

Q1: What would happen to your existing application when you add a new machine?
A1: Nothing – it wouldn’t even know that it exists if we wouldn’t tell it (through manual work).

Q2: Assuming that you addressed (1) – which part of your application would you run on that new machine?
A2: It depends... we need to measure and see  => meaning more manual work…

Q3: Assuming that you addressed (2) – what do you expect would be the impact of the new hardware capacity on your application, in terms of latency/throughput or concurrent users?
A3: We wouldn’t know until we measure it in real life => meaning lots of manual tuning, testing, optimization and in some cases redesigning your entire system.

The solution

The challenge I was trying to outline doesn’t necessarily point to a flaw in virtualization or even cloud computing, which is basically an outsourced version of virtualization. It is more to do with the fact that the IT world has applied the concept of virtualization only to the lower level part of the stack – the infrastructure, and expected that it will solve all its inefficiencies. Conceptually, I believe that virtualization is the way to go but it needs to be applied through the entire stack, as I outlined in one of my earlier posts (The Missing Piece in Cloud Computing – Middleware Virtualization). To learn how to apply the concept of virtualization through the entire stack, it must be better understood how virtualization works in other layers, above the infrastructure. If you examine different virtualization technologies such as storage virtualization, operating system virtualization and desktop virtualization, a pattern emerges:

The Virtualization pattern

1. Break big physical resources into smaller logical units

2. Decouple the application from the physical resources

3. Provide an abstraction that makes all the small units look like one big unit

Scaling pattern of a virtual resource

When you scale a virtualized resource, you basically plug in more small physical resources, and thus increase your capacity. The abstraction layer is responsible for detecting these new resources and adding the new resource to its pool. Since the application is decoupled from these resources, it “sees” the increased capacity without necessarily worrying about where those resources exist.

Making it more efficient through resource sharing and pooling

The way to make the solution more efficient, is to pool and share resources together among multiple instances of the application. This is often called multi-tenancy. The general idea is that you can pool the resources of multiple users of your application, and assume that none of them is going to require your full capacity, so you can put them on the same underlying hardware. Obviously, one of the the biggest challenges with multi-tenancy is isolation, i.e., how to let each user “feel” as if she is running on her own dedicated resource.

I know this is a fairly simplistic view of the concept, and obviously doing this for a mission critical application that is running in production is going to require much more thought. In part II of this post I’ll discuss in more depth how to apply those principles through the entire stack.

It's Complex to be Simple

Avdi Grimm wrote an interesting post Simplicity is Complicated. Avdi bring few interesting points on how the term Simplicity is often abused to justify all sort of decisions.

“Everything should be made as simple as possible, but not simpler” - Einstein 

The Meaning of Simplicity

“Simplicity” means different things to different people depending on the context of the problem you’re trying to solve. Avdi provides a useful typology of cases in which the term “simplicity” is often used (and abused): minimizing unnecessary effort, hiding complexity, avoiding difficult-to-understand features, avoiding formal architecture, elegance of design, staying close to the domain.

I’ll spare you the details but I would highly recommend reading the full post. One of the points that I tend to see quite often in various discussions these days is the use of simplicity to justify new language choice by measuring the number of lines of code, elegance as a complexity measurement:

“sometimes when you choose to do less you force someone else to do more. And sometimes that someone else is you, two weeks down the road”

In other words, reducing the number of lines of code (elegance of code) is only one measure of complexity. Other measures include existing skillset, learning curve, available toolset, maturity, interoperability with legacy and other third party tools, etc. Choosing a certain language because it will reduce the amount of code may seem to be logical for a specific area of our application, but at the same time may not be the right choice if you take a broader perspective and look at the entire application or project. At that level, introducing a new language can actually end up introducing huge complexity.

Scalability & Simplicity

Scalability is one of the areas in which I could see almost all sort of the simplicity arguments being abused to justify a certain architectural choice. I have often used Twitter as an illustration for this type of thinking. For example, one of the more critical decisions in any product is the selection of the application framework. Twitter was well known for its selection of Ruby. Simplicity (and productivity which is essentially a synonym of simplicity) was used as the main argument for selecting Ruby vs. Java or any other framework at the time. Twitter made a very similar simplicity argument to justify the selection of Java and Scala over Ruby a year later. It would seem that the term “simplicity” was used to justify a certain choice, and then the exact argument was made to justify almost the complete opposite choice. This is a great illustration of how the simplicity argument becomes a tool to justify almost any choice.

I’ve argued that “simplicity” was an excuse to bypass some basic methodical thinking that could have saved Twitter the painful trial-and-error process. For example, it was a well known fact that Ruby didn’t provide a good concurrency story. In addition, Ruby was lacking many of the performance tuning measurement tools that exist in Java. It shouldn’t be that hard to realize that if your looking for scalability, relying completely on Ruby is going to put some strong limits into your architecture. If you add this to the mix, the question arises: is Ruby really that “simple”? Avdi brings another interesting observation on that regard:

Ruby on Rails made web development simple, relative to tools that came before it. It accomplished this feat by embracing "convention over configuration". In order to do this, it had to incorporate a great deal of extra complexity in the form of algorithms which guess the intent of the programmer instead of forcing her to specify her wishes explicitly. Anyone using it eventually has to become familiar with these hidden rules in order to understand why Rails behaves the way it does.

In other words it is “simple” if your application falls directly under the realm of assumptions of those who designed that framework and could turn out to be complex if you fall outside the boundaries of that realm. So the answer is that you shouldn’t take it for granted that ROR is “simple”.

Another example I’d like to use in this context is GigaSpaces. With GigaSpaces we worked really hard to make scalability simple. Our mission was to enable application to run on a distributed cluster of machines as if it was a single machine. We achieved that by removing the number of moving parts (SBA) and by providing an end-to-end architecture that will solve the scalability of the entire application from the load-balancer to the database. Yet the consistent feedback I heard, at the time we first introduced this concept, was that GigaSpaces is complex. When I asked users for an explanation the answer was quite interesting. Most users defined their problem domain in terms of a database bottleneck. They didn’t define their problem in terms of end-to-end scalability, and they often weren’t even aware that solving only the database bottleneck is not going to make their entire application scalable.

When cloud computing emerged, it forced users to think about elasticity. Elasticity means that you expect that your entire application will scale up or down when a new machine joins or leave the network. Elasticity forced users to look at how they can scale their entire application. All of a sudden, GigaSpaces appeared to be a significantly simpler solution then most of the available alternatives, which relied on static application server clusters and tier-based implementations that didn’t quite fit the dynamic cloud model.

That brings us to another question: What makes the same product “complex” at one point in time and “simple” at another point of time?

Avdi provides an interesting answer to that question:

Complexity is like an air pocket trapped under a layer of airtight plastic. If you push it down it just pops up somewhere else. Most of our attempts to "simplify" software really amount to pushing the complexity somewhere (or some-when) else. And our disagreements about what is meant by simplicity are really about where we think the complexity belongs. ..

In other words it all depends on where you define your area of complexity when you’re making your technology selections. Changing the definition of complexity from “solving database bottleneck”, to “application elasticity” yields very different results.

Final Words

There are tons of examples I can think of where I could see how the same platform or product could turn out to be “simple” for a certain set of challenges and “complex” for other set of challenges. It’s important to note that even though I used Twitter and Ruby as illustrations, the arguments are not specific to those examples and can be applied to almost any language, framework or product be – be it PHP, Java, Spring or even GigaSpaces.

As Avdi pointed out, the goal of “simplicity” can be used to justify just about any decision and therefore we need to be very cautious when we, or somebody else, bring it up to justify a choice. Making a certain complex problem “simpler” is often not enough, if your users define their complexity in a different area. I’ll close with Avdi’s advice:

I challenge myself and anyone who reads this, next time you use the word "simple", to stop and think about what you mean by it. As an exercise, try to rephrase it in terms of where you are pushing the complexity.