Nati Shalom's Blog

Is Your Data Really Secured?

Security has always been one of those topics that we as developers or architects hate to deal with. Our ideal world would be that security is dealt with at some higher level in our data-center, and that we don't need to think about it. It turns out that some of the most recent frauds can be caused by someone from within our own organization. In this specific fraud case, the cost to SocGen was $7.7b.  There was nothing specific in this incident that was a direct result of an application security failure, but it does force us to examine more carefully some of the current assumptions about security. In this post I focus on some of the potential security vulnerability that we may be exposed to as a result of using data-grid and caching more and more within our organization. 

Why Do We Need a Secured Data-Grid?

We add the data-grid/caching layer as a front end to the database to address the database scalability limitation. One of the common assumptions in this sort of model, is that the caching layer is an embedded part of a particular application. In other words, the caching layer can only be accessed by a specific application. Based on that assumption, it is fair to assume that it is the application's responsibility from a security perspective, to control access to its caching layer. Therefore, security was not generally considered to be a critical aspect of many of the existing caching solutions.

However, there are two things that change that assumption:

• Caching becomes a shared service (just like databases of today), that is shared by more than one application. SOA-based applications are a good example of this. SaaS-based applications impose an even bigger challenge in this regard.
• Virtualization and the drive for better utilization, forces us to think of a more efficient deployment model through sharing of resources. Having a dedicated in-memory cluster per application can be a fairly inefficient model, but on the other hand, running in a shared environment requires a high degree of isolation support by the data-grid service.

Security assumptions and constraints change significantly when you move from a cluster dedicated to one application, to a cluster shared between several applications. It is no longer enough to assume that the application is responsible for granting access to its embedded middleware components. In this case, multiple applications which share the same data, must have the right level of isolation and access. Securing the shared middleware resources is the key to achieving this level of isolation and control.

In the next section I describe the specific attributes that a secured data-grid must support.

Securing Your Shared Data-Grid/Caching Layer

1. Granting Access Based on Roles and Content

Imagine the following cases:

• Several applications can access the same data-grid service, and be granted different permissions to view each data item, based on the content of the data.
• Different parts of the same application (e.g. purchasing module vs. sales module) may be granted access to different accounts, even though all the account data is stored in the same shared data-grid.
• One part of an application can be granted read/write access, while another part of an application can be granted read-only access to the same data.

This scenario is very similar to what we expect from most databases today. A user can be granted access (authentication) through user-name/password authentication, and they can be granted read/write access (authorization) to the shared data. Providing such a level of control is fairly basic and core to any secured data-grid. However, there are cases where we would want to grant access to different parts of organizations, or where customers need to have different privileges for different parts of the shared data. For example, in a call center application, we might have a shared call center service that serves different departments in our organization. In such a case, we would want callers of certain departments to see only the contacts that belong to them, and not be granted privileges to see or have access to contacts that belong to other departments. In that case we would need to provide fine-grained authorization, based on the actual content.

2. Ensuring Isolation in a Multi–Tenant Environment

There are many cases where we do not want to share the same data-grid between different groups of applications, but at the same time we do want to share the physical memory resources. A multi-tenant environment implies that we run multiple services or applications in a shared environment, but keep them logically separated, as if they were running in their own dedicated environment. That level of separation is referred to as isolation. In order to do that, we need an abstraction layer that enables us to provision our data-grid as a set of fine-grained services that can be managed independently, even if they run on the same process. Abstracting the data-grid as fine-grained services enables us to provision the data-grid services on a shared group of machines or even processes (JVM’s), and still manage them completely independently.

There are a few components that enable that level of fine-grained isolation:

• Life cycle management – each data-grid can be started or relocated, completely independently of other data-grid services, even if they run on the same process container.
• Class-loader  - each data grid contains its own class-loader to ensure that there is no class version conflict between two data-grid services.

In addition to that, there are other levels of isolation, such as ‘zones’ where we can ensure that certain data-grid services never share the same process or even the same machine. At the top level, we can assign a data-grid with specific virtual machines (VM), which ensures the highest level of isolation, not just at the service level, but also at the operating and physical resources level.

Providing this level of fine-grained isolation control between data-grid services that share the same process, up to a completely separate data-grid that runs on its own virtual machine, is critical if we want to ensure best utilization on the one hand, and isolation on the other hand. Providing a limited degree of isolation determines how effectively we can share resources.

3. Transport Level Security

In most data-grid deployment, it is fair to assume that network level security is a responsibility of the data-center, not the data-grid, or even the application. There are cases where the data-grid can span across multiple firewall or data-center zones, and have a replication channel between those zones. In such cases, the replication channel can turn out to be a security loophole. The simplest approach to address this challenge, is enable replication over SSL. Using SSL can ensure that no one can “sniff” the packets that go over the wire.

The Administrative Challenge

Providing all this level of fine-grained security control and multi-tenancy can pose a huge administrative challenge. For example:

1. A super administrator needs to have full access to all data-grid services

2. An application administrator only needs to be granted access to data-grid services that belong to a particular application.

3. Operational managers need to be granted monitoring (read-only) access, to view how the system is behaving, but don't need to be granted access to change the data-grid topology or configuration.

These are only few of the challenges that we have to deal with, once we move from a dedicated data-grid to a shared data-grid. This basically means that we can’t assume that all administration operations to our data-grid are the same.  We need to treat operation guys differently then we deal with application managers, and they all need to have visibility and some level of control to our shared services. That means that we have to rethink how to manage security when we deal with shared data-grid services.

Not Just Your Data..

The same arguments that I have mentioned here, apply not just to the data-grid, but to any part of the application middleware that is going to become a shared resource. For example, think of the messaging service, or any other part of your application that becomes a service to other applications. This implies that we need some sort of generic way to handle application security that is consistent across the layers of the application.

How Security Reduces Costs

Being able to share data-grid services between multiple applications is a great cost saver. Here is how:

1. Better utilization  - we can better utilize the machines that run our data. In other words, we reduce the total number of machines that run our data services. The fact that we use less machines to run our data-grid services means that can save all of the cost associated with it i.e. cooling, maintanance, space etc.

2. Pooling of spare resources – each data–grid needs to run some level of spare capacity. Sharing of data-grid services enables us to share that spare capacity, and therefore better utilize it amongst different applications.

3. Reduction of the amount of redundant data – one of the ways that organizations dealt with the lack of ability to share their data services effectively, was by maintaining different copies of that same data. That led to redundant copies, as well as a lot of synchronization cost associated with it. Being able to safely share the data, means that we remove a lot of that overhead, and the cost associated with it.

4. Reduced administrative overhead – the data-grid becomes just like a data-base: instead of having each application group learn how to provision it separately, and go through the cycles of tuning and sizing it separately, we now have all that knowledge and skill set managed by a shared group. This is the same way in which Amazon manages its SimpleDB, SQS and other services - we as users of those services don’t need to worry about how to provision or maintain it.

Final Words

Caching/data-grids are going through a similar evolution to databases. As with databases, we started by using caching as an embedded service to the application. Now we are in the phase where we need to be able to share the data between multiple applications, or in cases where we don’t want to share the data, we need to be able to share the resources for managing the data, while keeping a high degree of isolation. The demand for these sort of requirements becomes much more common with SOA or SaaS-based applications.

We used to think of security as a defensive move. However, security is not just about protecting our data – security can create the right level of isolation that is a main enabler for achieving better efficiency and cost savings, through sharing of resources, as I pointed out above.

As we approach the next generation of middleware and data-centers, it becomes clear that we cannot move to the next wave of virtualization and cloud computing without a strong security and isolation solution that is built-in to all layers of our application and middleware.

GigaSpaces Security as a Reference for a Secured Data-Grid:

If you are interested to learn how these challenges have been addressed specifically by GigaSpaces, you are welcome to join a technical webinar that we will be conducting on that subject. The webinar will take place on Wednesday, October 28, 2009 9:00 AM - 10:00 AM PDT

GigaSpaces and GoGrid: Cloud Platform for Enterprise Applications

I had few discussions in the past week with people who asked me about our recent GoGrid announcement and how our cloud offering integrates with the GoGrid environment. I’ll try to answer the most frequently asked questions below. In addition, we’re conducting a webinar where you can ask your specific questions online and watch a live demo – see details at the bottom of this post.

What is the GigaSpaces cloud offering?

GigaSpaces provides a set of cloud-enabled middleware services, namely data grid, messaging and Map/Reduce. These are equivalent to the Amazon SimpleDB, SQS and Map/Reduce, with the the main difference being that it’s implemented entirely in-memory and designed to support existing enterprise applications. On top of that GigaSpaces provides integration of those services into a Platform as a Service solution designed to make the deployment of JEE/Spring or XAP-based applications extremely simple.

What makes it enterprise ready?

Most enterprise applications are based on standard JEE or .Net platforms.  In addition, all enterprise applications are transactional in nature and cannot tolerate any potential inconsistency. Enterprise applications are more sensitive to performance and latency. Our cloud-enabled middleware supports all these requirements. It was designed to leverage memory resources to gain the best performance and latency, and supports large-scale deployments from the get-go. It is based on XAP, which is a mature product with hundreds of deployments in the most mission-critical applications.

What makes your middleware cloud-enabled?

GigaSpaces supports all the key characteristics expected from cloud-enabled middleware, as defined by Gartner below:

How is it different than traditional middleware?

With GigaSpaces all middleware services run on a shared pool of resources. GigaSpaces provides a high level of security to ensure complete isolation between multiple applications. The deployment, provisioning and monitoring is completely automated.

Can your middleware seamlessly shrink and grow dynamically when a machine joins or leave the network?

Yes – let me explain:

There are three components that makes our middleware elastic and thus suitable for a dynamic environment:

1. Service automation and administration – this part takes care of moving the middleware and application services around based on machine, memory, and CPU utilization.
2. Common clustering and state management – this part makes sure that nothing breaks when you start to move around any of the applications or middleware services.
3. API Facade – this part provides a standard interface as well as language binding, for seamless access from existing applications.

What does the integration with GoGrid include?

The integration includes the ability to use any of the middleware services (data-grid, messaging and Map/Reduce), either individually or integrated with our PaaS solution.

Can I use it if I'm not running on GigaSpaces XAP?

Yes. Users can leverage the GigaSpaces images to get deployment automation and improved management for their existing application, as well as add auto scaling and high availability. 

What is unique about the GoGrid cloud offering?

For GigaSpaces users, the main things that you’ll find useful in the GoGrid environment are as follows:

• Enable custom deployment – GoGrid enables enterprise users to customize their environment to fit their specific enterprise standard. This include VPN, network as well as machine hardware configuration if required.
• Native multicast support – makes cluster deployment simple.
• Standard database storage support – makes database high availability simple.

For GigaSpaces windows/.Net users, you’ll find the following useful:

• Built-in .Net support
• Support for Windows and SQL Server 2008
• GoGrid have the same pricing for 64/32 bit

When can I try the GoGrid integration?

The GigaSpaces/GoGrid solution is currently available for limited set of users. Public availability is expected within a month. If you’re interested in early access, or want to be notified when the images will be publicly available, please send an email to <cloud at gigaspaces dot com>.

Is there a live demonstration?

To see a live demonstration, you should register to the Enterprise-Grade Cloud Platform with XAP and GoGrid webinar that will take place on October 14 at 9:00 AM PDT.

Why Larry Don't Get It

Once again, Larry Ellison, CEO of Oracle, made a joke of Cloud computing, saying that "a cloud is water vapor."

Ellison made the same argument last year – Geve Perry wrote a nice piece on Ellison's Anti-Cloud Computing Rant back in September 28, 2008 (note that the timing in both cases happens to be fairly close to the Oracle World event).

Many of the arguments that Geva brought back then still apply today. Clearly, Larry has a vested interest to argue that “nothing is new under the Sun.”

I actually tend to agree with Larry – Cloud doesn’t bring any real new technology to the market.

But what Larry, and many folks that he represents, don’t get is that cloud is not about bringing a new technology to the market. Cloud is about making the the same concept that we have today much more efficient and simple so that it can be adopted by the masses. Everyone in this business knows that building technology for high-end enterprises is vastly different then building the same technology for small startups. But again, I completely understand why someone who wants to re-brand an old mainframe concept as a new “database machine,” and bought an entire hardware company to serve that purpose, would claim that there is nothing new under the “Sun” (no pun intended).

Anyway, I'll have to give Larry credit for being a real performer and hilarious in the way he articulates his arguments.

For those who missed the video here it is:

How Map/Reduce and the Cloud are Affecting Analytics Applications

Two weeks ago I gave a presentation at the Sun HPCW workshop in beautiful Regensburg, Germany. It was a great chance to meet many of the folks from the Sun Grid Engine team face to face.  It was also a great chance to take a step back and look at the evolution of one of the interesting areas in the industry - analytics. Even though this is one of the more technically challenging areas, I find it odd that relatively little has been discussed about this within the developers community. 

In this post I will try to provide an overview of the different approaches for running analytic applications. I will also examine how patterns like Map/Reduce and a new environment such as the Cloud, drive a new approach that is significantly simpler and more cost effective than many of the existing approaches. There are other models for analytics such as stream based processing or CEP that i chose not to cover in this post as they deserve represent a new paradigm that deserves a separate discussion.

What is Real-Time Analytics (a layman’s definition)?

Here is the simplest and shortest definition I could come up with (probably not the most accurate one, but good still good enough):

· Analytics: The process of manipulating raw data (a.k.a data mining) into a meaningful report.

· Real-Time Analytics: The ability to generate the desired report in near real time.

Analytics in the Finance Industry

It is interesting to note that financial applications within organizations are organized based on the “speed” with which they need to react to the market. The Front-Office for example, would be the trading/market data application that needs to react to the market event in a fraction of a millisecond. The Middle and Back-office is the part of the organization that looks at an aggregated view of the market and tries to draw various trends, based on the current market and history. The algorithm that is used to process that data and generate the required report is referred to as Monte Carlo simulation.   A typical set of reports generated in that model would be P&L (Profit & Loss), VAR (Value at Risk), Reconciliation etc.

In previous years, those reports were generated on a post trade basis, i.e. at the end of the trading day, the data from the current day was aggregated into files or a database, and during the night there was a set of services that processed the data and generated the desired reports. That meant that if there was an event during the trading day that could affect the portfolio, it would only be taken into account the next day. At the same time that the volume of data increased, the window of time to process that data (during the night) became shorter due to globalization (time zone differences shortened the “night”). This led to scenarios where simulation accuracy was limited by these margins.

Analytics Becomes Mainstream

In general you can categorize any application that looks at an aggregated view of a stream of data, as an analytic application. Fraud detection would probably be another good example of analytics, since it tries to continuously monitor user actions in order to determine whether their action is part of a fraud or not. Social networks, Ecommerce and even personalization, such as search engine personalization have become more mainstream. With that, the demand to process vast amounts of data to produce various market trends, user behavior, fraud behavior etc. becomes not just useful, but critical to the success of the business. A search engine would be a good example of that – the quality of the search results is a direct function of how well you are able to aggregate content and user search patterns, and apply it to the way you produce the search results.

The Evolution of Analytic Application Architecture

To make things simple, I will start with a very basic view of the analytical application architecture, which represents the two most common architectures:

Phase 1: First-Generation Analytical Applications

Phase 1-1: Excel spreadsheet

Everyone uses Excel to generate various reports, so the use of Excel and spreadsheets are probably the most common and popular way of generating reports. The way this is done is by loading the data from a file, and then using Excel to generate the graphs and various views of that data.

Phase 1-2: High-End ETL process

Excel works well for small data sets, but for large data sets a batch process is used that is based on ETL (Extract Transfer and Load). It represents a batch process in which the operational data from the production sites is normally aggregated into some sort of data warehouse. In the process of doing so, the data is stored in a format that is optimized for generating reports. The process normally involves extracting the data, transferring it into an optimized format, and loading it into the warehouse for further processing. The ETL process normally happens once per data source. The data processing (a.k.a data mining) is where most of the heavy duty work takes place. In many cases, once the relevant reports and analysis have been produced, the data becomes obsolete and at this point it is common to clear it and store it in a long term archive.

The Need for Real-Time Analytics

You don’t need to be a genius to realize that that if all reports could be generated in real time, this would probably be the ideal world. The only reason this is not done, is purely technical. No way had been found to aggregate all this vast amount of data and process it in real time. Traditional databases had to be tuned completely differently, to deal with real time transaction processing and complex analytics (type of locking, data structure, transactional semantics etc.).

“Real-time data … requires a continuous flow of data, and traditional enterprise data warehouse deployments introduce slow data delivery to BI environments” (Forrester)

The challenge became even worse as the volume of data increased exponentially, and at the same time there was a continuous pressure by businesses to process even more complex analysis at an even lower cost. It was clear that addressing those contracting requirements couldn’t be achieved by just “throwing more hardware” into the problem. That led to the next phase of analytic application architecture, i.e. a grid-based analytic system.

Phase 2: Second-Generation Grid-Based Analytical Applications

Phase 2-1: Speeding up Analytics via Parallel Processing

Phase 2 was not yet trying to get into real-time analytics but was mainly trying to find ways to speed up the process of analytics. The majority of the focus in that area was on the batch processing that in some cases could take days or hours. By introducing grid computing, the processing time could be brought down from days to hours, and from hours to minutes.

With grid computing, the way that processing time is speeded up, is simply by spreading the processing into a large pool of commodity hardware. By splitting a linear process into small chunks that can be processed in parallel, the processing time can be speeded up quite significantly.

Phase 2-2: Adding an In-Memory Data Grid

As the time to process the data went down from hours to minutes, other factors that were previously considered insignificant became much more significant. It became clear that to speed up the analytics beyond the current level, closer to real time, it was necessary to consider ways to speed up the load and data access time, because they became more significant relative to the processing time.

“Real time drives database virtualization - Database virtualization will enable real-time business intelligence through a memory grid that permeates an infrastructure at all levels" (Forrester)

This approach became fairly common during the past few years. There is more information about this approach in one of my previous posts:  Bringing Data Awareness to the Grid

Challenges with Grid-Based Analytical Systems

While technically the combination of Compute and Data Grid were able to bring the time it takes to perform even very complex analytics to near real-time, the adoption of that solution was still marginal. One of the main reasons that this solution was not widely adopted was cost and complexity. Only a few organizations could afford building and maintaining large data centers. The skill set for running and building applications that could run in those environments was fairly rare, and therefore the labor cost was also rather high.

Phase 3: Next-Generation Public/Private Cloud-Based Analytics

The cloud makes a grid-based system affordable. So with the cloud it is possible to reduce the hardware cost, and more importantly it avoids the huge initial investment. The fact that the cloud is more widely adopted also reduces the labor cost and ramp-up time required to build such systems.

Having said that, the fact that anyone, even the smallest startup can have a large data center at a distance, reduces not just the initial investment and the direct hardware cost, but opens up a new set of opportunities for building such applications. For example it is possible to outsource the entire analytic process to someone else and use it as a service.

This was the case with Primatics Financial that provides risk analysis for mortgage portfolios as a service.

The Cloud and Map/Reduce Make Analytics Simple

With grid-based systems that run in the data center, the pool of machines is fixed. One of the main roles of the grid scheduler is to manage the pool of physical resources which belong to that pool. The cloud on the other hand already takes care of most of the resource scheduling part. Virtualization makes it possible to get the isolation required to maintain the separation between multiple jobs. If these two factors can be leveraged, a simpler model can be developed.

One of the ways to simplify the analytic application is to simplify the way that parallelization is handled. Since most of the resource scheduling is dealt with through the Cloud/Virtualization layer, a simple pattern such as Master/Worker, Map/Reduce can be used to parallelize, i.e. to distribute the application across the available machines.

You can read more about the Map/Reduce, Master/Worker approach here, and how it compares with a grid-based system.

Reducing the Number of Moving Parts

Another way to simplify and optimize the application is to reduce the number of moving parts.

Rather than having a different set of services to load the data, hold it in memory, and yet another set of services that computes the analytics, it is possible to bundle the loader, the data and the processing services into one application unit, and have multiple instances of those units running to handle the scale. In this case, each unit handles the entire life cycle (the loading, processing etc.) of the data analytics food chain.

SaaS Enablement through Multitenancy Support

One of the core elements that is required to provide the analytics application as a service is multitenancy. Multitenancy is the ability to run the multiple analytics processes on a shared infrastructure as if they were running on dedicated resources. To add multitenancy to our service, the life cycle (Start, Stop etc.) of each of the analytics needs to be managed independently of the other analytical processes that are running over the network.

There are two possible layers of isolation that can be chosen:

Service-Level Isolation

Service-level isolation provides isolation at the application component level. You can deploy multiple services on the same process/machine; you can run different versions of that same service in the shared environment and maintain them in complete isolation to other services that are running in the same environment.

Since all services share the same machine, they can potentially access shared resources such as the file system, etc. It is also harder to control things like CPU utilization and memory allocation on a per service level.

Machine-Level Isolation

Machine-level isolation is provided through the use of a virtual machine. In this case, each application or group of applications runs on separate virtual machines. Virtual machines provide an additional level of isolation that includes the actual CPU, Disk and Memory resources.

Combining Service-Level and Machine-Level Isolation

Both service-level and machine-level isolation are actually complementary approaches, as each one of them provide a different level of isolation and application packaging granularity. Service-level virtualization tends to be more fine grained, but at the same time provides a lower level of isolation. Therefore, it makes sense to use it in areas where real CPU, Memory and file system isolation are not that critical, but getting better utilization from the existing machines is more important, and vice-versa.

In the diagram above, you can choose to run everything from the same organization at a service level of isolation, and use a VM level of isolation between different customers to ensure that each customer’s application is running in a completely private set of VMs.

Private/Public Cloud

With the introduction of private cloud providers such as VMware with their Vcloud and Vsphere approach, and open source alternatives such as Eucalyptus, you can choose to run the same solution in a hosted environment or locally. You can also use a hybrid approach where some of the processing runs in your local environment and the rest in the hosted environment. Things like the Amazon Virtual Private cloud make such a hybrid deployment model fairly simple, as the local and hosted machine are accessed in the exact same way.

Final Words

The continued demand for processing a growing amount of data in real-time and at a lower cost, drives continuous innovation in the way data is processed.  Grid-based analytics (Compute and Data) addressed the demand for scaling, and also reduced processing time quite significantly, but at a relatively high cost and complexity. The demand for real time analytics by mainstream applications in the social networking media created a drive for a simpler and less costly solution. Cloud computing enables the cost of hardware to be reduced, but at the same time opens up new opportunities for making the analytic processing significantly simpler. You can rely on the cloud for scheduling and managing your resources, and on virtualization for multitenancy. With that, a simplified parallel processing approach, such as Map/Reduce, can be used to handle the parallelization of the application.

“it's quite clear that the business-intelligence/data-warehouse industry is moving toward a new paradigm wherein the optimal data-persistence model will be provisioned automatically to each node based on its deployment role -- and in which data will be written to whatever blend of virtualized memory and disk best suits applications' real-time requirements." (Forrester)

The Interactive Cloud (Part II - Adding Operational Awareness to your Application)

In part I of of this topic, I covered the general concept and motivation behind the "Interactive Cloud". I described how the interactive cloud enables applications to be managed in a similar way to the way we run our business. In this post, I discuss further details of what it means to add operational awareness to the application. To do that, I refer to our specific experience in GigaSpaces. Towards the end of the post, I point out specific code snippets as an illustration of that model.

Background

GigaSpaces 7.0 Application Cluster Management API is an API for managing application clusters, and for giving applications operational awareness. The reason why we came up with this API in the first place, is quite interesting. When we started GigaSpaces, a typical deployment was primary-backup with multiple clients connected to this pair. Sooner then we anticipated, we found ourselves supporting cluster deployments of 10s and 100s of nodes per application. We are now at a point where customers are starting to use GigaSpaces XAP as a service shared between multiple applications, which in itself increases the deployment size quite significantly.

In this type of environment, troubleshooting a production system can be a real nightmare for everyone involved. We started to add all sorts of metrics and logging information into our system but it hardly improved things. The process of finding the right information at the level we needed was still a heavy-duty task.

When we looked back and tried to analyze how we learn about the application’s behavior, we saw that quite often we identified the time when things started to go wrong, and then, using the timestamps on the different operations, ran a complex correlation process to see what happened on each machine during that time. In many cases when we hit the first area of trouble, we were still missing information, because we were running without fine-grained logging, for performance reasons. So then we would turn up the log level and wait for the next event, and so forth.

We realized that just adding more probes and log levels and using some sort of CEP (Complex Event Processing) to parse all the garbage we were going to generate, on a continuous basis, was not going to be good enough. Even worse, if the reporting system got so complex, we would eventually end up going back to the old logs because the reporting system would inevitably lose some of our events :)

We figured that we could not afford to continue down that path – the cost, both on our end and on the customer’s end, was way too high. We needed to try a totally new approach.

The Ideal Solution – Getting Help from the Applications themselves

When we analyzed what would be the ideal process, we came to the realization that the application should be responsible for the monitoring. When the application starts to behave abnormally (for example, CPU spikes, GC spikes), or experiences network failure, it should do two things:

1. Take corrective action – the application should figure out if the problem is that resources have been exhausted. If that is the case, it should try to rebalance itself to reduce the load automatically.
2. Provide information to help analyze the root cause of the event that occurred – the application should point out the machines that are suspects and ask those server instances to increase their logging level and print a snapshot of thread dump. This provides only the information that is actually needed to perform root cause analysis, and at the right level of granularity.

It wasn’t surprising that some of the root cause analysis indicated that one of the biggest contributors to failures, is human error or misconfiguration. That led us to the realization that the way to reduce those factors would be through complete automation of manual deployment and maintenance processes.

Another thing we realized during that process, is that when things went wrong, quite often the operation guys became fairly useless at a fairly early stage of the process, and the application guys had to be called in to understand what happened. Not surprisingly, the application guys didn’t even know how to get the required information or didn’t have permission to get that information, and had to rely on someone else to do that for them – which made the entire process really long and complex.

All of this led us to the realization that we needed to create an application cluster management and monitoring tool that could be used by both the application developers and the operation guys. The application guys could use that API as part of their development cycle, and the operation guys could use it to receive the right level of information into their management and control dashboards.

In the following section, I provide a short sneak peek into this API, and how it is structured.

XAP’s New Application Cluster Management and Monitoring API (a.k.a. the Administration API)

There are basically two ways to design  the API for this challenge: define the interaction model and drive the API from there, or define the domain model and drive the interaction model from the domain model. We chose the latter.

1. Defining the Application Management Domain Model

The first step was to define the domain model of our application management. The diagram below maps out the “nervous system” of an application running on XAP – it shows the core layers of the application management domain model.

 

2. Defining the Interaction Model – the API

Once we mapped the nerves and sensors of our application, we could start to wire them. This enabled us not only to monitor the state of the application components, but also to interact with them when we needed to take corrective actions (see snippet below).

3. Making Operational Awareness an Integral Part of the Development Framework

If we want the application to take more responsibility for its operational behavior, we can’t just point it to an external management and monitoring tool that doesn’t speak the “developer language”.  We need to make the proactive and reporting behavior an integral part of the development framework. This is illustrated in the examples below.

Application Code that uses the Cluster Management API - a Few Examples

Below are a few examples that show how applications interact with the new Administration API, using a groovy-based API. These snippets were written in Groovy but they can be be called in Java as well.

Example 1 – Getting Statistics

This example uses Groovy closures to get callbacks when a new container or machine joins the network, and when a specific application event happens.

Example 2 – Calculating Average Load on a Web Container

This code snippet shows how to get an aggregated result of the total number of requests/sec from all the web containers in the cluster.

Example 3 – Scaling Up:

The code snippet below shows the process of scaling out, when the number of requests exceeds the maximum number of requests allowed per instance.

Note that the call to incrementInstance() generates an entire match-making process behind the scene. What happens is that a new web container is spawned, but the provisioning of this new container is based on the SLA that was defined at deployment time. In our case, it is deployed only on machines that don’t already run a web container and have spare CPU and memory capacity.

Initial Success Stories

Since we came up with the API, we started to see its use spreading in areas we didn’t even think of. Here are a few examples:

• Helping customers reproduce problem scenarios – now that we can script almost every part of the application, we can easily script even very complex scenarios, where the application starts to execute a few things and then crashes only when a certain operation happens, etc. The fact that we can script these scenarios makes it easier to create a reproduction scenario, and from there, the road to spotting the problem and fixing it becomes significantly shorter (see detailed reference here).
• Third part integration – one of our partners created a fairly extensible Eclipse plug-in. The reason he was able to do that pretty much by himself, was because the API gave him everything he needed in order to control the application development and testing lifecycle (see more details in Jeroen Remmerswaal post here).
• Writing a complex SLA – we were able to easily write complex SLAs, for example, automating deployment of applications across multiple data centers and making sure that primaries and backups never run on the same data center. All that without any human intervention.

What’s Coming Next?

One of the immediate things we are going to enhance, is to provide more built-in SLAs, and a set of utilities that perform deployment and implementation of mainstream scenarios, such as auto-scaling of web applications and auto re-balancing of data grid deployments. This is going to serve as a cornerstone of our new cloud-enabled middleware initiative, where all the classic, enterprise-grade middleware components become available as a service. In fact, some of our customers are already using our In-Memory Data Grid as a shared service between multiple applications within the organization. The idea is to make that type of deployment simpler and intuitive.

Take Part in this New Initiative

I can go on with more of my thoughts on what should be done, but I’d rather stop here and ask all of you for your feedback. If you’ve faced similar challenges, how do you see this solution? What would you want to see in this new package?

For those who want to try out this new API, it is worth noting that it is actually provided for free as part of our Community Edition, so anyone can simply download it and try out the examples from this post.

The Interactive Cloud (Part I)

When we think about cloud computing, the first thing that comes to mind is on-demand computing. Indeed, cloud computing and virtualization providers made it easy to get a machine by just providing a credit card and making an API call. However, that being said, what happens after you get that pool of machines? How do you deploy the application into these machines? How do you know how the application behaves? How can you react when something breaks or when the load on the system gets too high?

In the human world, many organizations face pretty much the same challenges. It is fairly easy to get a good infrastructure for business – office space, electricity, telephones, computers – but as a manager, it’s really difficult to place the right people in those offices, give them sufficient work but not so much that they’re totally overloaded, track what they’re doing, and react properly when something goes wrong. A recent and very successful methodology for dealing with this challenge, which is quickly gaining popularity, is called Agile, Scrum,  or Lean. The common ground of this family of methodologies is that unlike the centralized management approach, we work through short iterations and continuously correct the course of our actions based on the actual progress. We rely heavily on the individuals to take responsibility for their actions and be part of the decision-making process. We assume that those individuals will take their own corrective actions for things that fall under their direct responsibility. The program manager is only responsible to make sure that the course (and budget) is kept on the right target and shift resources or allocate new ones if needed.

Today’s Operations and Applications – Just Like Centralized Management

Chances are that you are aware of the Agile methodology and recognize that it solves many of the huge difficulties of traditional, centralized management. But at the same time, if you think about it, you are still running your applications and IT operations according to the defunct, centralized management paradigm. We’ve invested years in creating an artificial separation between what is known as “operations” and “applications”. In this way of thinking, “applications” is responsible for maintaining a certain business logic and “operations” is responsible for monitoring and maintaining the application. Operations management is responsible for monitoring the CPU, memory and every individual aspect of the network configuration, application configuration, and consumption of resources. The application is not really aware of any of this, and usually you rely on the operations guys to take corrective action, or at least give you a call and let you know, when something out of the ordinary happens.

This seems natural because it’s a pattern we’ve all become accustomed to. But just imagine managing a real life project (with people) using this same approach. It would take the centralized management approach to frightening extremes. Here’s how it would work – there is the project manager (operations) and the employees (application), with absolutely no direct communication between them. Each employee has a set of sensors attached to her, which report on health conditions (heartbeat, blood pressure, and so on), and another set of sensors that report how quickly she is working. The project manager views all this data in a nice dashboard with nice graphs and all sorts of reporting schemes. Whenever there is an alert – for example, one of the employees is running a fever, or is working very slowly for some reason, the project manager troubleshoots the issue and sees how to solve it and whether it affects the project.

That’s ridiculous, isn’t it!?

In real life, especially when working with Agile, employees are responsible for their output. If they don’t feel well, or are working slower than usual, only the individual employee can really know what the problem is – or whether there is a problem at all. They take a rest or figure out what else they can do about it. They only report back to the project manager if they really can’t deal with the issue themselves and it’s starting to affect their ability to meet the project goals. The manager doesn’t need to constantly monitor their health or throughput – the only thing worth monitoring is if they can or can’t do their job. The reason is less of interest. More importantly, the manager relies on the employees to report when such an event happens, and to take corrective action themselves.

If that’s so obvious, why can’t we apply the same same common sense to the way we run our software? Why can’t we make our applications aware of their condition and able to take some responsibility for themselves? This is important in any computing environment, but I would argue that it is especially important on a cloud or in any virtualized environment, in which the operations/application paradigm totally breaks down.

The Solution: Bringing Operational Awareness to your Applications

Like a real-life employee, the application needs to be responsible for its own health. It needs to report only on general status and any shortage of resources. The operation guys don’t need to continuously monitor the health condition of each individual application component. They should rely on the applications themselves to send the right warning signals before something breaks, and suggest corrective actions in a language that the operation guys can understand. For example: “I need X amount of additional resources ” or “I can release Y amount of resources after I've done my duty.” Deciding which specific task needs to be assigned to each of the available resources should be the application’s responsibility – in any case, it probably knows best how to utilize these resources. 

To make this a reality, you need to add operational awareness to your applications. By operational awareness I mean that the application needs to be written with the assumption that it can be asked to be moved at any point in time, and the application needs to take responsibility for events that can be solved at the application level, and take corrective actions to meet its SLA. The manager should only be notified in cases where the application has exhausted its physical resources and there are no other available resources. For example, if one of the instances of the application runs out of memory or needs more CPU power, and there are spare resources that are not utilized by other resources within the application, it is the application’s responsibility to balance its resources within its own realm. But even if there are no available resources, the manager doesn’t need to bother with the fine-grained details of how to balance the individual application resources. Balancing these resources should be kept within the scope of the application itself. The manager should only bother to allocate the pool of resources between the various applications.

Learning from Amazon EC2

A working example of this approach is the Amazon EC2 API. In my view, one of the revolutionary concepts that made EC2  so successful, was the realization that rather then creating high level tools for managing the data center, Amazon chose to provide an API and let the application developers figure out how to run their application. The API opened up a whole ecosystem of third party solutions, and a whole new market. What’s interesting is that in EC2, when I want to auto-scale my application, I don’t call the operations guys to do that. There is a very slim chance that someone who doesn’t understand my application would be able to write such a policy without breaking the application. Most of the applications that run on EC2 just call the EC2 API when they want to spawn a new machine or manage their high availability, or auto-scaling of their application. The nice thing about the API is that it doesn’t prevent Amazon and others coming up with more high-level tools that are much more useful for the operation guys at a later stage. The main difference is that the application takes more responsibility for ensuring that it meets its SLA, and the operations guys get the type of information that makes sense to them, and more importantly, something that they can easily act upon and take the appropriate corrective actions.

The Need for an Application Cluster Management API

Amazon provides a comprehensive set of APIs that work at the Virtual Machine level. Back to our real-life project analogy, the Amazon tool provides us with the real-estate  - the office space, the network, the computers. What we don’t get is the access to the individuals that do the actual work. For this reason, keeping the API at the virtual machine level is often not granular enough, as it doesn’t enable us to interact with the individual application components, manage and automate the application’s deployment lifecycle, define its dependencies and distribution model. There are other frameworks such as Capistrano that became popular, mostly in managing Ruby deployments. This type of tool is becoming popular in other cloud deployment settings as well. Capistrano can address the application deployment and basic life cycle management, and provide a good foundation for automation of those processes.

While the combination of the two is a good starting point, it is lacking the notion of application cluster. In an application cluster, you want to have more explicit semantic support for managing deployment topologies (partitioning, replicated). You need built-in support for defining backup and primary, and you need to be aware of the data affinity requirement of certain elements in the architecture. Without all this, enabling the application to re-balance its resources when a new machine joins or leaves the network, is close to impossible.

The Interactive Cloud

The combination of image-level provisioning and an application cluster management API, leads to a new form of cloud deployment experience that I refer to as the interactive cloud.

The interactive cloud operates fairly closely to the way we manage human-led projects. The image API is equivalent to the physical resources that are critical to run our project, e.g. the office, the computer, and the network. Another part of the project is managing the individuals that run the project. Learning the right way to manage our teams, from the lessons of Agile project management, is through a close iterative process. This is where the Application Cluster Administration API comes into play. This API gives the application developer the ability to control and automate application deployment, and continuously balance the way the application runs, to ensure that it meets its required SLA. It also provides a callback mechanism when you run out of resources and when some of the idle resources can be released. You can also easily generate an aggregated view for operations (without going down to the level of what each individual component did at each given point in time).

With this combination, you can spawn new applications, automate their deployment and lifecycle, start new machines as needed and keep a close watch on your application behavior after it has been deployed. When a new machine joins the cluster, the application automatically rebalances its resources to take advantage of that new machine, and when machines fail or are removed from the cluster, the application rebalances itself on the smaller resource pool. If something breaks, the application automatically scales down, allowing it to continue working despite the failure, and in the background allocate an alternate resource to fit in instead of the new one.

IMO, a cloud/virtualized environment forces us to go in this direction. This is an area where the old way of separating “applications” from “operations” is just not going to cut it.

Final Words

There is nothing really new about the solution I propose here, and the concept behind it. Quite the contrary. The interactive cloud allows you to apply proven methodologies that work well for many critical projects in the real world, and apply them to the way you run your applications. Most people who run their application in the cloud have already started to build and manage their applications in this way, using tools like the Amazon EC2 API and Capistrano. However, this was mostly done as an afterthought and was mainly driven by intuition and common sense. My attempt in this post is to back up these initial efforts with a clear methodology.

In my next post (Adding operational awareness to your application), I will use the Administration and Monitoring API, new in GigaSpaces XAP 7.0, as an example for such a framework. It is worth noting at this point that this API is actually provided for free as part of our Community Edition, so anyone can simply download it and try out the examples I will provide in the next post. I’m looking forward to showing that this vision of the interactive cloud is really very tangible and is already at everyone’s fingertips.

The Word is Out on GigaSpaces XAP 7.0

GigaSpaces XAP 7.0 was released in mid-July. For those who missed the buzz, in this post I’ll try to provide a summary of the main perspectives that were voiced about the new release. At the end of this post is a list of additional articles and sources for those interested.

Gartner – cloud-enabled XTP, greater appeal to mainstream enterprises

Massimo Pezzini from Gartner wrote a detailed review, GigaSpaces Focuses on Cloud-Enabled XTP With XAP 7.0, where he covered the main points of the release as follows:

Version 7 introduces several technical improvements aimed at better supporting cloud deployments, including:

• Full service-level-agreement-driven autoscaling of the platform — now extended to a Java EE Web container (currently Jetty, though others will be added)
• New administration application programming interfaces, extending the XAP management tooling, that enable integration with third-party monitoring environments and make it possible for XAP applications to “automanage” themselves (for example, triggering a new virtual machine in a cloud environment to increase processing capacity)
• Significant performance improvement and a simplified programming model for multicore processor environments
• Metadata-driven, multitenancy support for cloud deployments
• Improved integration with the Amazon Elastic Compute Cloud (EC2)

GigaSpaces' progressive and well-established support for popular standards like Spring and parts of Java EE is already helping lower users’ concerns about use of an “unusual” platform. More user support will likely come from the planned adoption of OSGi Alliance standards in future versions. The full integration of Jetty is a notable new marketing proposition that can appeal to mainstream enterprises looking for ways to non-intrusively scale up and down Java ServerPages/Servlet applications. This paves the way for more strategic adoption of XAP full capabilities in these accounts. The core DCP, the cloud deployment options and the new multitenancy capabilities are likely to appeal to independent software vendors, system integrators (SIs) and leading-edge organizations looking for new technology opportunities for business diversification.

Wall Street & Technology Journal – private clouds, commodity hardware, faster time to market

Penny Crosman from the Wall Street & Technology Journal provided a fairly detailed overview on who is using GigaSpaces XAP on wall street:

Wall Street clients are using XAP to create private clouds, allowing applications to share resources. One firm is using it for applications that support real-time portfolio analysis of profit and loss. Another is using an XAP-powered private cloud for a reconciliation service; prior to XAP, each application at this firm had a dedicated infrastructure. To prepare for peaks and lows, they had to provision many hardware units. Today, two applications share the same infrastructure, and the firm is creating infrastructure as a service.

A market data distributor is planning to go live with XAP 7.0 in August to provide certain online services. The company will use XAP to better allocate resources and better scale and integrate the web tier of applications.

One big financial organization had a high-end hardware infrastructure that cost over $1 million per year in annual maintenance. The company switched to XAP, invested $300,000 in commodity hardware, and achieved an annual savings of $946,000 in hardware maintenance alone, because of the fact that they're running on simple commodity hardware, according to Bar Sadeh.

Another big financial service organization estimated the cost to redesign a customer support application using open source tools at $425,000, just for development time and resources. "They didn't bother to add the expenses of memory and hardware because this cost estimate already exceeded what they would have to pay for XAP," Bar Sadeh recalls. The firm also estimated a nine-month effort for the project, however, using the GigaSpaces software it took four months, she says.

DZone interview – all the patterns you need to build scalable applications, working out-of-the-box on a single platform

I had numerous interviews describing the new release. I’d like to point out the one that was published on Dzone specifically: 

XAP 7.0 offers all the patterns you need to build scalable applications in a simple and cost-effective way – based on lessons learned from our demanding enterprise customers and from the new generation of Internet services such as Google, Amazon and Twitter. These patterns include asynchronous even-driven design, partitioning/shards, parallel execution and Map/Reduce, consistency without distributed transactions, and moving the database to the background while persisting all data with 100% reliability and consistency (more details on all of these in my blog). XAP 7.0 offers all these patterns working out-of-the box, on a single platform, without forcing the user to shop around for different packages and point solutions, figure out how to glue them together, and take the risk that the weakest link in this complex chain will break the entire application.

You can read the full story here

eBay Marplaats.nl – map/reduce for simple aggregated search, massive hardware savings

This was followed by joint webinar with Cees de Groot, Lead Architect of eBay Marktplaats.nl; I'll probably cover the Markplaats project in more detail in a future post. Cees discussed how they used the patterns I mentioned above to scale their hugely popular classifieds site (over 6.4 million unique visitors per month) and how he used patterns like Map/Reduce to execute a simple aggregated search. Cees also mentioned why they ended up choosing GigaSpaces over Terracotta. He discussed their challenges with their current PHP application and how they where able to take a gradual approach to overcome those challenges with XAP. One of the interesting outcomes of this process is the fact that they were able to reduce the number of servers in their data center by at least 65%, through the use of XAP’s in-memory clustering and by employing a combination of advanced scalability patterns.

I highly recommend listening to Cees talk – even as someone who was closely familiar with this project, I find that I learn a thing or two every time that I hear him discuss it (Cees’s talk starts 20 minutes into the session).

Shay Bannon and Uri Cohen – distributed autowiring, custom SLAs, multi-core performance boost

I would also recommend looking into Shay Banon's blog post, Distributed Autowiring with GigaSpaces Executors, where he provides a good technical overview of Map/Reduce, a common pattern for executing complex aggregated queries, made famous by Google and used in many other scenarios, including social networking sites.

Banon also provides a good technical overview, together with Uri Cohen, of a real-life usage scenario of Map/Reduce, in a webinar which took place last week. The webinar also covers other areas of the release, such as the new administration API and how it helps users write their own SLA.

I can't help but show a slide from the webinar presentation, which shows the performance improvement we've achieved in a multi-core environment. I'm really proud of this. Here it is:

Summary and future Road map

As a summary, I'd like to quote Massimo from Gartner, who concluded his review of the 7.0 release as follows:

Other vendors, including Oracle and IBM, plan to extend their DCPs with OSGi, Spring and Java EE elements. But none has yet architected a full cloud-enabled XTPP like GigaSpaces. This gives the company at least a year's competitive advantage on the technology front.

In future releases, we plan to continue in the same direction, bringing the value of virtualization and cloud computing to our enterprise users. This will include better support for data center integration, extended integration with production monitoring, built-in SLA and deployment automation, and of course, additional improvements to our In-Memory Data Grid in terms of performance and resource utilization. At the same time, we'll continue our efforts to make the product simpler and reduce the learning curve, by adding more support for standard APIs like JPA. Stay tuned…

References cited above, and a few others

• Release Notes - What's New in GigaSpaces 7.0.0

• GigaSpaces Focuses on Cloud-Enabled XTP With XAP 7.0 (Gartner)

• GigaSpaces Unveils Cloud-Oriented App Server (Wall Street & Technology)

• Nati Shalom talks about XAP 7.0 (DZone)

• Hyper-Scale Cloud Apps with GigaSpaces XAP (CloudBzz)

• How the New XAP 7.0 Admin API Helped Us Troubleshoot Burning Issues (blog by Kobi Kisos, our QA team leader)

• UI & Manageability Improvements in XAP 7.0 (blog by Uri Cohen)

• Distributed Autowiring with GigaSpaces Executors (blog by Shay Bannon)

• GigaSpaces XAP 7, The “Real Option” for the Cloud and On-Premise (blog by James Liddle)

• GigaSpaces 7.0 Release (blog by Guy Nirpaz)

From static to elastic web applications

During the past few weeks I gave few talks about our latest XAP release and some of the new models that we introduce in this new release. During all those talks I found myself explaining what is the difference between what I refer to as a Cloud/SaaS enabled application platform, which is basically what XAP is, and traditional application platforms, and how XAP enables taking an existing static application and turning it into an elastic application easily.

In this post I’ll try to provide a summary that explains what are the motivations for this concept, and the different pieces that comprise it in the XAP Cloud/SaaS enabled platform.

The motivation

The introduction of virtualization and cloud computing (Public or Private) enables us to create a new machine with a single API call. This is an extremely powerful capability that can change the way we design and deploy applications. Instead of running an application on a static set of machines, you can shrink and grow the application as needed.

The reality

The fact that we can now add machines using an API call is indeed an important milestone, however most of the existing applications in the data center are not designed to take advantage of this new capability. The reason is that they are tied together in so many areas, that any attempt at stretching them even a little bit will break the entire application.

Making our application elastic

To make your application completely elastic, you’ll need to get to the point where all the different pieces of the application can shrink and grow, while the application is running, and while at the same time ensuring that no data gets lost and the application continues to function as if nothing happened.

In the diagram below I'll try to sketch the various pieces that enabled us to provide a high level of end-to-end application elasticity with the latest XAP 7.0 release.

 

Common SLA driven clustering

In many of the existing tier-based products, clustering is dealt with separately for the web, messaging, application-server, and database tier. Each tier maintains its own set of assumptions regarding configuration and implementation. That in itself is a huge barrier to achieve end-to-end elasticity, because it turns each tier into a silo. This makes it extremely hard to move all those pieces together when something happens. In addition, the fact that there are so many tiers in our system makes the application extremely inefficient.

With that in mind, the first thing you need to introduce is a common clustering layer that can be shared across all the components in the system. By doing that, you reduce the number of moving parts, achieve consistent behavior across the entire application, and more.

There are basically two parts that comprise an application cluster:

• SLA cluster management – this the piece responsible for match-making between application components and the available resources. It monitors application behavior, making sure that it meets your performance, latency and scaling goals.
• Cluster state management – this is the piece that takes care that when we move pieces of our application around in order the meet a certain SLA, nothing gets broken and no data is lost.

SLA cluster management

At GigaSpaces we refer to this layer as “SLA-driven containers”. The SLA-driven container is basically a light-weight process that can wrap any application and report the status of the contained application. The Container Manager is responsible for maintaining the SLA of a given application when a failover or scaling event occurs. It does so by matchmaking between the required SLA and the available SLA-driven container resources. The SLA-driven container is written in a generic way, so it can wrap and manage other containers as well. A good example is Jetty, Glassfish, Spring and Mule in the Java world. Interestingly enough, this set of containers can also host non-Java applications. Currently that includes .Net and C++ applications. Once we get to this level of abstraction, you can easily support almost any application in the new, dynamic environment.

With the SLA-driven container concept, you can start any application in various deployment topologies: partitioning, partitioning with backup, replicated, etc. For stateless applications, you’d probably use only partitioning. For the stateful part of an application you’d use partitioning with backup, or a replicated topology. You can define the maximum and minimum number of instances that should be running at any time, assign the scaling policy that will trigger scaling up or down, and define a high availability policy to ensure that minimum number of instances is always running as long as there are enough machines available. With 7.0, we added support for “zones” – with a zone can group a cluster of containers into sub groups and apply policy at the group level. A good example is using zone between disaster recovery and primary sites. This ensures that the node in our cluster will run on separate zones.

Cluster state management

Cluster state management holds the state of the various pieces of our application in a reliable fashion. In most traditional systems, this piece was covered by a centralized database of some sort, or another file-system based solution. In a distributed and dynamic environment, this type of solution can be fairly limited, because having a centralized database to manage all your state information creates a huge performance burden and places strict limits on scalability. These limitations led to the implementation of proprietary solutions per product or per layer, which made the problem even worse, because now the state of the application is spread between different state management systems. Ensuring the consistency and reliability of the system in such an environment becomes extremely hard. Trying to dynamically scale and move pieces of the application is nearly impossible under these conditions.

This is where an In-Memory Data Grid (IMDG) becomes fairly handy. You can store the state of our all the various pieces of the application on top of the same IMDG cluster, and by doing that, remove the scaling and performance limitation that exist with the database or file-based approach. In addition, the In-Memory Data Grid was born to run in such a dynamically-scalable environment, and therefore it doesn’t inherit the limitations of a centralized implementation. In addition, the fact that you can manage state in one, shared location significantly reduces the number of moving parts in the application, which in turn enables running the application on much less hardware.

Managing the cluster through API

Clustering in most of today’s platforms is treated as a “black box” or even “black magic”. It is kept pretty much closed, and often very little is known about the implementation behind it. The only way you can control the clustering behavior is through a configuration file, which tends to be pretty fixed to the time of deployment and cannot be changed at runtime. In a virtualized environment, these types of assumptions start to break as it is very hard to anticipate what should be the “right” behavior beforehand and in addition, the impact of a failure, or a scaling event of one component in our system tends to effect other parts of our system in totally unpredictable ways.

XAP 7.0 comes with a brand new set of cluster management APIs, which are aimed to address this specific need. This is a revolutionary model in the application platform space, and so far I believe that XAP is the only product that provides such an API.  Personally I feel we’re only starting to scratch the surface with what can be done with this new framework. The most obvious part is the ability to monitor the system at a fine-grained level; the other part is the ability to write custom SLA and deployment automation tools. By automation I mean automating the deployment of an application at deployment time, but in addition, ensuring that the application is kept balanced in the way it uses the resources at runtime.

Putting the two together

There are various systems where state and application SLA management is implemented by two different technologies and products. That approach is doable and  can be a good starting point for moving into a dynamically scalable environment. However, in reality, you will soon find that there is a very strong dependency between the two. The decision on which pieces to move and where to move them, will in many cases depend on where the application state lives. For example, you would want to ensure that a backup instance won’t run on the same host as its primary, or in some other cases, the same data center as its primary. You’d want to make sure that the application is spread evenly around the different resources. Deciding on how to spread the application is dependent on the how granularly you can spread the data, etc.

One of the things we did in our product to overcome this limitation is by adding data awareness to our SLA-driven clustering and SLA-awareness to the way we manage our data. By this I mean that we provide built-in SLA for managing state and business logic dependency, as well as transaction support. On the data level we added specific semantics and call-backs to manage the lifecycle of our data instances at runtime. We also use a consistent id assignment for each data instance that ensure each instance contains the appropriate set of data, even if it changes its physical location.

By putting these two tightly together, we can safely manage the dynamic scaling of a stateful transactional application, and move pieces around when needed, while the application is running!

Virtualizing the entire application

Once you get to the point where you’ve managed to create a shared SLA-driven clustering infrastructure, you can start integrating that model into the specific application layers. Below I propose a gradual integration that can make it simpler to move existing applications into a dynamically-scalable environment, with zero or minimum changes to the code.

Web layer virtualization

The web-tier consists of a load-balancer and web-container. The load balancer is a gateway to the outside world and acts as a smart router of requests that arrive from the user’s browser to our web container. In most of the existing systems today, a web container is configured with a set of static IP addresses that form a fixed pool of resources for the application. In this way, you can load-balance the web traffic between this fixed set of resources. In stateful web applications, you would also use a load-balancing policy called “sticky session” to ensure that the state of a particular user session is kept consistent between various user requests.

Sticky sessions places limits on your application scalability, because the user request is routed to a fixed server regardless of the load on that server. In addition, if you add new servers to meet the load, only new user sessions can be load-balanced to these new servers.

To make the web layer completely virtualized, you’ll first need to remove the dependency on a fixed set of machines. Instead, you’ll use a dynamic discovery protocol to detect the availability of new web containers as they come and remove them as they go away. To reduce the dependency on sticky sessions, you can keep the user session on an external In-Memory Data Grid. In this way, the user session will be shared with all the web containers. Since the IMDG will be external to every request, this will involve a network call to load the session into local memory. To reduce this overhead, you can use “local cache” configuration; the local cache holds frequently-used data items and therefore allows you to load the user session only once and avoid the extra network call overhead.

Below is a more detailed description of the components from the above diagram

Dynamic load-balancer

The dynamic load-balancer is responsible for listening to availability of new web containers and updating the load-balancer when a new container joins or leaves the network.

Managed Jetty web container and session on top of the space

The managed web container is a wrapper on top of the common available web containers (currently we support Jetty as the main container). It enables taking an existing web deployment package (WAR package) and deploy it as-is on our web cluster. In this way you can easily turn any web container into a completely virtualized cluster without changing the web application.

You can also choose the web container of choice at deployment time.

Business logic and data virtualization

In most of the existing application platforms, there is a different package that deals with application business logic and data. In a JEE environment, the business logic is covered by a different set of APIs and implementation such as SessionBeans and  MessageDrivenBean. Spring provides a more light-weight version referred to as Remoting which is now part of EJB3.x. You can also use other RPC layers such as SOAP, or use JMS explicitly if you want to implement the business logic in an event-driven manner.

The data layer is often treated as a separate layer. In the JEE world, this layer is mapped through Hibernate or JPA. To manage the scalability of the data-layer, distributed caching is often used to reduce the database contention. Caching can be plugged into Hibernate (this is referred to as a 2nd level cache) or provided as an explicit layer that the application interacts with directly.

In a distributed environment, quite often there is dependency between the business logic and the data layer. This is referred to as data affinity. The number of moving parts, different packages, and different clustering implementations per package that existing in most of today’s platforms makes it quite hard to virtualize this part of the application,  let alone ensure data affinity.

As with the web layer, the key to virtualizing this layer is mapping both the data and all the business logic components into the same underlying clustering architecture. In other words, all the different APIs will be just a facade that exposes specific application semantics on top of the same clustering implementation. Since the business logic and the data layer is going to share the same high availability and data routing logic, data affinity will be built into the model. In other words, there would be no need for an external mapping layer to route a specific business request to where the data is.

Async persistency

The database is often a centralized piece in any architecture. It is also a contention point that can lead to devastating results in terms of scalability. At the same time removing the database completely can lead to a huge change in our application.

One of the common practices to enable database scalability is by front-ending the database with In-Memory Data Grid which is responsible for loading the data from the database into memory and keeping the database in sync by sending every batch of updates back to the database in an asynchronous fashion.

In this way, the user doesn’t need to block till the request is synched with the database and therefore doesn’t suffer the performance penalty imposed by the database; this eliminates the database’s contention and the scalability limitations it creates.

Taking a step by step approach:

For “green field,” taking the approach I outlined above makes the most sense. However, in reality you need to assume that:

1. You can’t afford a complete re-write

2. You have dependencies on existing pieces of the application that cannot be moved into this model easily.

The diagram below shows how to take a step-by-step approach. The idea is to start with things that wouldn't require changes to the code but should still provide substantial value. The best starting point would be scaling the web-tier, and then adding session high availability and scalability, all the while keeping the database and other pieces of the application as is. You can then scale the application business logic with little changes to configuration – this involves replacing the existing SessionBean and Remoting services with a virtualized implementation of the same services.  

Note that we you can keep database scaling as the last piece in the architecture, because scaling the data layer is going to require changes to way our data and application is architected.

Grow as you need vs. rewrite as you need

There are basically two common approaches toward complete application scaling that I've seen so far.

• Rewrite as you need – don’t worry about scaling at the beginning, and pay the cost only when you need to. The built-in assumption here is that many applications will never get to the point where they will ever need to worry about scaling, and it’s better to avoid optimizing things that you might never need in the future. Having said that, and based on many of the hero stories around Twitter, Facebook, eBay and many others who became successful, they were all forced to go through a very painful process of re-write when they reached that point. The cost of that rewrite can be much higher then the actual development cost, because this is a sensitive time when you might your market to your competition. One of the stories I heard about Facebook is that it owes its success to the period of consistent failures experienced by MySpace.

• Grow as you need –   indeed, at the first stages of the project, you don’t necessarily want to spend time optimizing things that you may never need. But the assumption is that the optimization requires lots of effort. As noted above, the step-by-step approach gives you the option to balance effort vs. value at each step. The effort would be the same as with any alternative approach. The main benefit though, is that when you need to grow, you won’t need to go through a complete re-write. The transition can be fairly smooth. In addition, you don’t need to switch and shop for different solutions – you already have everything built into the system to support your growth.

As I wrote in a previous post, Platform as a Service: The Next Generation Application Server? – the built-in assumption that scaling is complex is mostly derived from the fact that most of the current platforms were not designed for scaling. By contrast, the new generation of platforms are designed for scalability from the ground up. When you write your application in Google App Engine or Salesforce, you are not really exposed to all the details of the infrastructure. Most of those details are completely abstracted from your code. True, you need to know what’s going on behind the scenes if you want to reach the best performance and avoid potential consistency issues, but that is already a completely different level of effort than what we are used to with existing systems.

An analogy I like to use to describe this change is multi-threaded programming. Before Java, developing multi-threaded applications was fairly rare. Most applications where written in a single-threaded manner. When Java built threading into the language, writing multi-threaded applications became the norm rather then the exception. The same thing happens when we move to the cloud and virtualized environment. Writing a dynamically scalable application becomes the native way to write an application rather then the exception.

References

• Platform as a Service: The Next Generation Application Server?

• GigaSpaces Focuses on Cloud-Enabled XTP With XAP 7.0

• Scalable as Google, Simple as Spring - Presentation from SpringONE

• Hyper-Scale Cloud Apps with GigaSpaces XAP

• How the New XAP 7.0 Admin API Helped Us Troubleshoot Burning Issues

Platform as a Service: The Next Generation Application Server?

Platform as a Service is a term that can be fairly confusing for many people. Normally the term is associated with Google App Engine from Google and Force.com from Salesforce.com as the main references for this model. From a technical point of view, it is aimed to provide a similar type of value to the one that is currently provided by many of the application servers i.e. it provides a generic container that can host different applications and shield them from the details of the underlying operating system, network, database implementation. Unlike most of the existing application servers it was designed for massive scaling from day one. Another big difference is in the way it is being consumed. With PaaS you don’t need to install any software and go through all the hoops to setup a cluster environment etc. PaaS is provided as a hosted service that is pre-configured and installed. You get a production ready environment right at the start.

Key Characteristics of a Cloud/SaaS Enabled Application Platform

Last week I came across David Mitchel Smith presentation from Gartner. David provided a good definition to the main PaaS characteristics. A snippet from his presentation covering this area is given below:

What’s interesting is the great emphasize on Multitenancy support. The fact that the platform is going to be shared between multiple applications and potentially even different customers require various levels of tenancy support and isolation to ensure that even though were taking advantage of the fact that we can share resources between applications, each application needs to be able to use the platform as if it is running on its own dedicated resources. Another interesting point is the need for XTP support. Many would view XTP as a niche that is normally referenced in the high end part of the market, so it is fair to ask why would XTP fit into a general PaaS solution?  XTP represents a model for supporting enterprise transaction processing applications in an extremely scalable environment. In our case, scalability is not necessarily driven from the demand of a particular application but from the fact that many applications are going to run on a shared environment.  A PaaS targeted to enterprise applications would need to provide support for this level of scalable transaction processing support as a core service.

Can you run your existing business applications with GAE or Force.com?

No. Unfortunately as in anything in life reality can really spoil the party.

Force.com was designed to make it simple to run business applications that are database centric i.e. CRM, Reporting etc. It provides a rich set of high level services that make building such applications extremely simple. Google provides a more generic application platform and recently announced support for Java which is a big step towards reducing vendor lock-in concerns.  Google seems to be geared for consumer based applications. Force.com offers a more high level platform that is based on its proprietary services. This means that in order to take advantage of their service you will need to go through a complete re-write.

Force.com is based on a database centric architecture. They also seem to be limited in scalability as they partition their database per application. This means that if your application needs to scale more than what a single database can provide, you can find yourself pretty much locked.

Google's recent support for Java makes their offering closer to standard JEE application servers, however their current support impose a lot of limitations due to their sandbox model. These limitations mean that at the end of the day GAE can be applicable only to a small set of relatively simple applications.  Since there is no guarantee or control over the resources that you are going to receive from their underlying infrastructure, it is likely that the application performance will be unpredictable and will therefore be affected by other applications that are sharing the same hardware.

The fact that the platform as a service shields you from the details of the underlying infrastructure is what makes it simple and that is both the advantage and limitation. You can’t control the environment, you can’t choose your operating system, you can’t install your own set of services and you can’t control the performance characteristics of this platform.

This puts a huge adoption barrier for most current enterprises.

IaaS vs PaaS

Infrastructure as a Service provider provides a hosted service model that offers plain machine level access. This model is also known as Server as a Service. The fact that you get access to the bare metal allows you to run almost any application on this hosted environment. Unlike PaaS, IaaS gives you extreme flexibility. You can choose your own operating system, install any package that you want, you can setup your firewall, security etc. Amazon is known to be the leader in this space and also provides  a set of services on top of their infrastructure such as SimpleDB, SQS and MapReduce. Having said all that, this flexibility comes at the cost of complexity. In many cases you will need to install your own software, configure it, tune it etc. before you could make it run effectively on the cloud. Many application developers don’t have the skill-set to do that. This exposes many operational challenges as many organizations are not geared to support this type of environment from their own IT. The high level services provided by Amazon are still proprietary and would require a complete re-write if you plan to use such services.

PaaS for Enterprise applications – doing it right

The ideal solution would be to combine the best of the two worlds i.e. the flexibility of IaaS and simplicity of PaaS, and here is how:

• Build a Generic PaaS on top of AWS – To build a PaaS we don’t need to re-invent the wheel. Unlike Google and Force.com we don’t need to own the infrastructure, we can actually use Amazon infrastructure or even better build our PaaS such that it can be portable between Amazon IaaS and a VMware IaaS. By doing so, the PaaS can provide us the ability to deploy applications in a simple way just as in GAE but would still enable us to control the environment, install our own software and get the full flexibility that IaaS provides.
• JEE as first class citizen – Many of the existing enterprise applications are built in JEE. Making JEE a first class citizen within our PaaS environment will enable you to leverage the existing skill-sets within those organizations. Similar to the standard model that has more than one implementation out there, reducing the lock-in factor significantly.
• Pre-configured for extreme scalability – All the services provided through the PaaS will need to be implemented and pre-configured for extreme scaling and come with a production ready setup to enable dynamic scaling and fault-tolerance.

Next generation application Server?

Yes, In my opinion, PaaS represents the next wave in middleware technology. One that is targeted for virtualized enterprises, one that was designed for scale-out from the get go, one that fits the new way of delivering SaaS applications and one that can be extremely simple to use. The current PaaS players i.e Google and Salesforce and to a lesser degree Microsoft represent one type of player, those that own it all, i.e. the infrastructure and the platform. There is already a new emerging category of players in the PaaS market, Application PaaS players. Application PaaS players would specialize on delivering only the platform and not the hardware and network infrastructure. They act as the bridge that will enable portability between different infrastructure providers including the internal IT (a.k.a private-cloud). The application PaaS players will also be segmented in similar ways to the way application servers are segmented today, i.e there would be the one targeting the low-end consumer market in similar ways to Google App Engine, the ones that will be aimed toward the high end of the market and those who will be specialized in certain languages or development framework, i.e. Ruby/ Java/ .Net etc. In the PaaS type of world, those who would be able to provide a holistic solution that works smoothly across all the application tiers would have an advantage over those who are providing point solutions.

What about my existing applications ?

Most people would categories PaaS as a platform that is delivered through the internet. That statement would make the idea behind PaaS irrelevant for a large part of the existing enterprise applications, as a majority of them are not ready to run their application in a hosted service over the internet. Let’s examine that statement:

Many of the existing IT run farms of application servers in their internal IT. Those application servers are running in an internal data center that is not that different from any other hosted services, only that it is a specialized hosted service tailored for the needs of the specific organization. I would therefore argue that those organizations that are already running such application server farms would find it easier to evolve such server farms to PaaS model than to change their entire IT infrastructure into internal cloud. The reason is that those applications were already written to run in an application container model, therefore a large part of the transition work can be done within the container implementation and outside the application code. Targeting them first would therefore be potentially easier transition than trying to transform all your other applications into a virtualized environment.

I know that there are many people out there that would argue that internal PaaS is not a PaaS because it doesn’t answer the exact definition of PaaS. However in my view the similarities exceed the differences and the value to the enterprise would be almost identical to the one that I would receive by any of the internet based PaaS platform.

Will I lose control?

PaaS will provide the internal IT much better control over the applications that are running on their environment. You can have high visibility as to ways the application consume resources. In the same way, you can control fairly tightly the application security, scalability, resource management and fault-tolerance which can finally be managed in a consistent way across all the applications.

Final words

Platform as a Service represent the next generation application server IMO. GAE and Force.com are most known references in the market for that model. Both tries to offer the complete stack and that’s both their advantage and limitation. There already is a new category of Application PaaS providers that specialize on providing PaaS on top of existing infrastructure providers. There would be generic PaaS providers (similar to GAE) geared for the low end part of the market and those that are geared for the high end of the market. There would also be more vertical PaaS providers (Similar to Force.com)  i.e. those that will provide PaaS for a certain segment of applications or segment of the market such as Online Gaming PaaS, Telco PaaS etc. A good example for such a service is Twilio. As I outlined in my recent post (Google App Engine plus Amazon AWS: Best of both worlds), this is not just a theory but a reality in the making. GigaSpaces' contribution to this new reality is our new cloud framework on Amazon EC2. Geva Perry provides an excellent overview of other good examples that follows that same line his post What's Really Exciting About Cloud Computing.

References:

• Google App Engine plus Amazon AWS: Best of both worlds
• Waiting to see if App Engine or an alternative gets there first
• What's Really Exciting About Cloud Computing.
• JavaSpaces and the Next Big Thing
• Space-Based Architecture for AppEngine
• GigaSpaces as Alternative to GoogleAppEngine for the Enterprise
• GigaSpaces introduces portable PaaS
• Google Brings App Engine's Pros and Cons to Java
• Google App Engine and The Java Web; The Wrong Java?

No to SQL? Anti-database movement gains steam – My Take

Eric Lai published a provoking article on Computerworld magazine titled “No to SQL? Anti-database movement gains steam” where he pointed to many references in which different Internet-based companies chose an alternative approach to the traditional  SQL database. The write-up was driven from the the inaugural get-together of the burgeoning NoSQL community who seem to represent a growing Anti-SQL database movement.

Quoting Jon Travis from this article:

Relational databases give you too much. They force you to twist your object data to fit a RDBMS [relational database management system],

The article points to specific examples that led different companies such as Google, Amazon, Facebook to choose an alternative approach. I outlined below what i found to be the main drivers behind that trend:

• Demand for extremely large scale:

“BigTable, is used by local search engine Zvents Inc. to write 1 billion cells of data per day.”

• Complexity and cost of setting up database clusters:

“PC clusters can be easily and cheaply expanded without the complexity and cost of ‘sharding,’ which involves cutting up databases into multiple tables to run on large clusters or grids.”

• Compromising reliability for better performance:

“There are different scenarios where applications would be willing to compromise reliability for better performance. A good example for such a scenario is HTTP Session data. In such a scenario the data needs to be shared between various web servers but since the data is transient in nature (it goes away when the user logs off) there is no need to store it in persistent storage.”

Having said all that, it seems that many still agree that despite all the limitations of traditional database solution, SQL database are probably not going away:

“It's true that [NoSQL] aren't relevant right now to mainstream enterprises," Oskarsson said, "but that might change one to two years down the line.”

The current "one size fit it all" databases thinking was and is wrong

The article seem to point to an interesting trend where a growing number of application scenarios cannot be addressed with a traditional database approach. This realization is actually not that new. In  2007 I wrote a summary of Michael Stonebrake’s article, "One size fits all: A concept whose time has come and gone" on my blog: Putting the database where it belongs. The great thing is that It looks like this “old” news is spreading to the larger community. This can be explained by the continuous growth of data volumes, together with the growing need to process larger amounts of data in a shorter time. These two trends force many users to think of an alternative approach to the traditional database. The classic early adopters are those who hit the wall. It is very likely that as these alternative solutions mature, they will find their way into mainstream development as well.

Not your mom and dad’s database

The article seems to over glorify some of the alternatives that where mentioned while downplaying their limitations. A good example is Amazon SimpleDB. I wrote in the past a post about this, Amazon SimpleDB is not a Database, where I outlined some of the limitations of the Amazon SimpleDB solution. As you can see from these limitations, SimpleDB cannot and shouldn’t be positioned as a direct alternative to your existing database.

While I share many of the thoughts and enthusiasm of the anti-SQL movement, I would highly recommend taking very cautious steps toward any of the alternative solutions. It is very important that you make yourself familiar with their strengths and weaknesses, and avoid hitting their limitations at a point in time when you have very little room to maneuver. I’ve seen several cases where users developed their data model in a centralized model and expected that it will scale seamlessly once they switch to a partitioned topology. The fact that you can switch between centralized and partitioned topologies without changing your code doesn’t mean that your application will behave correctly and will scale as you expect.

This topic has actually been the center of a discussion in Architect Summit meeting we had last summer, which was hosted by eBay:

Abstractions and Partition Awareness
A horizontally-partitioned system typically provides an abstraction that makes the partitions appear as a single logical unit. eBay and Flickr, for example, both use a proxy layer to route requests by a key to the appropriate partition, and applications are unaware of the details of the partition scheme. There was near-universal agreement, however, that this abstraction cannot insulate the application from the reality that there partitioning and distribution is involved. The spectrum of failures within a network is entirely different from failures within a single machine. The application needs to be made aware of latency, distributed failures, etc., so that it has enough information to make the correct context-specific decision about what to do. The fact that the system is distributed leaks through the abstraction.

My recommendation would be that you design your data model to fit into a partitioned environment even if during the initial stage you’re still going to use a single centralized server. This will allow you to scale when you need to, without going to through a massive change.

What about in-memory alternatives?

An option that i found missing from this article and becomes fairly popular with many large websites is the use of an in-memory data store, In-Memory-Data-Grids as they are often called, such as Memcached, GigaSpaces, Coherence, eXtremeScale etc. With this model we front-end the database with an in-memory cluster which becomes the system of record and uses the SQL database as the background persistent store. For those looking to build social network graphs, real time events (as in Twitter), real time analytics, fraud detection, session management, etc., that is probably the more natural choice. Todd Hoff from highscalability.com wrote a very good article on this subject: Are Cloud Based Memory Architectures the Next Big Thing? I also wrote a detailed description how this approach works with GigaSpaces and MySQL: Scaling Out MySQL.

What about ACID transactions, consistency etc?

The traditional 2PC (two phase commit) model in which consistency is achieved through a central transaction coordination server is not going to fit with many of the distributed data management alternatives. In an earlier post, “Lessons from Pat Helland: Life Beyond Distributed Transactions,” Pat Helland suggested an alternative model to distribute transactions, a workflow model. Instead of executing a long transaction and blocking until everything is committed, break the operation into small individual steps where each step can fail or succeed individually. By breaking the transaction into small steps, it is easier to ensure that each step can be resolved within a single partition, thus avoiding all the network overhead associated with coordinating a transaction across multiple partitions. This has been one of the core concepts in designing scalable applications with Space Based Architecture (SBA). The Actor model that was introduced with new functional languages like Scala and Erlang is built into the SBA model, with the difference that in SBA, actors can share state and pass events by references, and thus avoid the overhead of copying the data with every transaction.

Shay Banon wrote a good description on how the Actor model works with SBA in this post:

“The above is a diagram of a simple simple polling container that wraps a service (Actor, the Order service in our example). The polling container takes (removes) events (Data), process them, and writes back Data to the collocated Space (Data Grid) it is running with.”

The need for real time data processing – a real life example

In the past weeks we had been involved with a prospect who is looking to add a social network to his eCommerce site. One of the requirement for this new service was the need to build a graph that includes friends of friends, and products in catalogs. The process for building that graph with an In-Memory-Data-Grid took 2-3 msec vs. tens of seconds with a traditional approach (note that part of the complexity in this case is that the query itself is ad-hoc and can’t be easily partitioned). Building that graph on-the-fly at these speeds just couldn't be done with traditional SQL database.

The reason that enabled us to get to this level of performance was:

1. We kept the data in-memory.
2. A large part of the complex query was pushed to where the data is (you can think of it as a modern alternative to stored procedure).
3. We used partitioning to spread the data and leverage the accumulated memory capacity of those memory instances.
4. We used both a scale-out and scale-up model to parallelize the query against all instances and take full advantage of multi-core as well as multi-machine power.
5. We reduced the number of network hops by pushing the the heavy data manipulation to where the data is and by returning only the accumulated result over the network.

Final words

In summary I would say:

• SQL databases are not going away anytime soon.
• The current "one size fit it all" databases thinking was and is wrong.
• There is definitely a place for a more a more specialized data management solutions alongside traditional SQL databases.

The adoption of these new solutions would be very much determined by two main factors:

1. How well they integrate with the “existing SQL world.”
2. How easy it will be to develop for these new alternatives, and how smooth a transition a given solution can offer for the average developer.

This is an area of continuous innovation that has been keeping us fairly busy in the past few years, and will probably continue to keep us busy. I’ll leave the details on how we deal with these two challenges to a separate post.

References:

• Architecture Summit 2008 - Write up
• Are Cloud Based Memory Architectures the Next Big Thing? 
• Scaling Out MySQL
• Actor Model and Data Grids
• Lessons from Pat Helland: Life Beyond Distributed Transactions
• Xtreme Transaction Processing under SBA terminology

• Social Media Kills the Database

• Amazon SimpleDB is not a DataBase!

Google App Engine plus Amazon AWS: Best of both worlds

George Lawton wrote a a good summary of my JavaOne talk in his article titled Google App Engine plus Amazon AWS: Best of both worlds 

Google App Engine (GAE) is focused on making development easy, but limits your options. Amazon Web Services is focused on making development flexible, but complicates the development process. Real enterprise applications require both of these paradigms to achieve success… What we really want is the flexibility and performance of AWS and the simplicity and ease of use of GAE.

This is exactly what we had been working on for the past year, leading us to the launch of our new cloud platform. With this platform we leverage GigaSpaces XAP as the high performance scale-out application server and Amazon as the robust and flexible IaaS. Together they form an alternative Platform as a Service geared for enterprise grade applications. This allows the cloud environment to inherit the extreme performance, latency and scalability of the XAP platform, which in turn enables achieving your performance and scaling target with less machines, implying a lower cost.

Real-life case study: Primatics financial – Risk analysis as a service

Francis de la Cruz and Argyn Kuketayev from Primatics Financial joined me through the presentation. In their part of the session they described their experience in developing a SaaS application for Real Time analytics.

Kuketayev described how Primatics used this approach to create a new automatically scaling cloud version of an existing banking application. Primatics initially developed a mortgage securities application that allows banks to estimate the value of a basket of hundreds of thousands of loans. The value of these loans fluctuates as economic conditions change and some portion of home owners cannot afford to make payments on their loans. Banks normally only need to assess the value of these loans at the end of each month, making them an ideal candidate for cloud services like AWS.

From a scalability perspective the challenge is to be able to provide a highly multi-tenant application that need to serve many firms, many users in that same firm each running many jobs at the same time. Implementing such a model can be fairly complex as you will need to be able to manage the life cycle of each job and each user independently and in isolation from one another.

 

Trying to build such a service directly on Amazon is going to be fairly complex, as you can learn from George’s summary below:

Primatics wrote the first version of EVOLV:Risk as a hosted web application for a regional bank.. The application needed to be fault tolerant so that if one node crashed, they did not have to restart the application over again from the beginning. Kuketayev said that it is not just about the loss of four hours, but the office is trying to close out the month and needs to access data to end the monthly cycle so they can go home.

Using GigaSpaces' toolset they rewrote the entire application infrastructure in about four-months to run on top of AWS. Now they can kick off as many instances as required for different banking customers, and each instance runs significantly faster than before. Kuketayev said that it is important for banks that none of their applications run on the same infrastructure as another bank.

The diagram below shows the specific architecture that Primatics ended up using. Those that are familiar with Space Based Architecture would find it fairly straight forward:

The application is built out of  a set of processing units. Each processing unit contains the compute agents in the form of a polling-container.  The compute agents gets a a reference to a remote Data Grid that is shared by all processing units. Each agent gets the job injected to it by the polling container and gets a reference to the data it required to process the job. Once the job is completed, the result is stored back in the space. The results are flashed out back to a database through a mirror service.

In a case of a failure, other compute agents are able to continue from the exact point of failure and continue the job processing as if nothing happened. This is because the state of the job is kept safe in the data-grid and not in the agent’s memory.

Kuketayev from Primatics nicely summarized thye lesson he learned after going through the experience of trying to build it on his own vs. trying to use GigaSpaces:

Kuketayev said that one of the biggest lessons is that you need to have your infrastructure do the provisioning for you automatically, or otherwise you end up spending a lot of time just turning things on and off. He said they are now using configuration APIs to automate this process, whereas before they were using scripts. This allow for automatically throttling and failover recovery without human intervention.

Kuketayev advised "You need to make sure you use the right tools … You don't want to have to worry about provisioning and reliability. Make sure you have provisioning, failover, monitoring and SLA out of the box."

The full JavaOne presentation is available here:

Alternative to Google App Engine on Amazon

View more OpenOffice presentations from Nati Shalom.

Final words

Fr solution providers the size of Primatics, building a risk analysis application as a service couldn’t be possible without cloud computing. Cloud enabled them to offer their solution as a service without the need to go through major investment of building a data center to support it.

Primatics’ experience is not special. One of the benefits of building Software as a Service is that you have one shared environment for all your customers. At the same time, one of the challenges is that in a shared environment, failure becomes more public and will impact ALL your clients. If the system doesn’t scale well, you’re going to be hit twice as hard as in a standalone application.

Building a robust and scalable SaaS application can be fairly complex. A good cloud infrastructure will get you a first class data center, but it won’t solve your application requirements.What’s interesting with cloud computing is that it forces you to think about the cost and efficiency of your application more than ever before. In the Primatics example, running a simulation of 100 nodes for 3 hours is very likely to fail at some point. A failure during such a simulation will immediately cost you 300 hours, not to mention the fact that you might lose the simulation window for the day and the reputation challenge you’ll will be facing with your customers. In addition, putting the data in-memory and making the application run 3-5 times faster means that you would need 1/5 of the machine power, which saves 80% of the cost of running the application.

I believe that the challenges imposed by cloud computing force us to focus on what we do best and avoid investing in areas which are not core to our business. Because the pay-per-use model significantly lowers the cost barrier, going down the path of writing your own infrastructure, as many have tried to do before, will be much more expensive and risky then ever before.

References:

• Google App Engine plus Amazon AWS: Best of both worlds 
• The Need for Speed - Saving Costs By Improving Application Efficiency
• eBay Subsidiary Uses XAP to Boost Performance - Expects to Meet Same Loads with 95% Less Servers
• The impact of cloud computing on build vs. buy behavior

GigaSpaces Launches a New Version of its Cloud Computing Framework

We have been working for a while on our new Cloud Computing Framework (CCF). We wanted to wait until we had some real customers and partners behind the platform in production before spreading the word on a larger scale. And now, I am very happy to report that we have reached this point sooner than expected and just in time for the JavaOne event.

The new version includes an enhanced integration with the Amazon billing system Devpay designed to enable users to use GigaSpaces as a service on a true pay per use model. All you need is an Amazon account and your done. Unlike other products, that are offered in similar fashion, we decided not to provide just a bundle of our software packed in an Amazon image (AMI) but instead provide a more tight integration with EC2 that will enable our users to use a single GigaSpaces product for all GigaSpaces versions as well as customize the installation to include their own software packages and application code on the fly. The new CCF provides a true end to end experience that will enable our users to deploy and manage their *entire* application from the load-balancer to the database in one single click.  The latest release of CCF includes support for the new GigaSpaces 7.0 release. Users that are looking to try out the new XAP product can now do that for free without the need to download or install anything.  This will enable you to try out any of the built-in examples, test the high availability features and run benchmarks in just one click. The new version includes lots of additional improvements to the user experience as well as new features and bug fixes. To view a detailed list of the new features click here.

Those that will be visiting JavaOne this week will be able to join the hands-on labs session and experience a full deployment of JEE application using CCF. You will also be able to gain insight from one of our customers, Primatics Financial on how they used CCF to build their Risk Management application as a service – See  full details here

In this post I will try to address some of the most common questions that we often receive when introducing this new service:

Who is using GigaSpaces CCF and Why?

The new release is already used by different customers in the following areas:

ISV’s and Service providers using CCF for SaaS enablement:

Many ISV's are looking for ways to move their standalone applications into a SaaS based offering. In doing so they face a few challenges:

1. Scalability – now their application needs to serve not just the users of a certain customer at a certain location but all customers from all locations. This increases the scalability challenge on their product by an order of magnitude.

2. Continuous high availability – one of the challenges that is imposed by having a shared platform for all customers is that the impact of failure can be catastrophic as failure can affect all of your clients at the same time and becomes public fairly quickly.

3. Portability – There are cases where some customers would still require their application be installed in their local data center for security and latency purposes. Today, this often means you will need to develop two seperate branches of your product to serve these two scenarios. With GigaSpaces you can write your application once and deploy it in the cloud or in your customer data center without having to modify your application and without locking your application to a specific cloud provider.

4. Performance – In developing high performance applications such as Trading, Risk Analysis, or SIP server, keeping the latency at sub msec and throughput at a maximum becomes critical. This is where our XAP middleware has a proven record in running numerious production mission critical applications.

The following are a few public references that chose to use XAP:

- Primatics Financial is using GigaSpaces CCF to build a high performance risk management solution

- Orbyte is using GigaSpaces CCF for building state of the art high performance trading applications that can be offered as service.

Other customers include Online Gaming companies that were looking for a simple and cost effective manner to launch new online games and provide a gaming application platform that enables other gaming providers to host their applications easily in a cloud environment. See further details on our online gaming solution here.

Enterprise customers using CCF to improve time to market of new applications and for large scale testing

According to recent research conducted by Forrester, one of the main requirements driving cloud adoption by the enterprise market is that current IT can’t keep up with the business demands of their organization:

• Capacity planning is too difficult – The current method of provisioning applications based on estimated peak loads has become either too costly or impossible. Most systems are non linearly scalable which means that knowing how many resources would be needed to meet certain goal is known only through trial and error.
• Time to market – launching new applications and services often takes months and sometimes year.
• The business wants a simple and cost effective way to prototype – not all applications make it to production but these initiatives still require the business to invest upfront in the purchase and configuraiton of infrastructure to support these prototypes.
• Cost saving – There are sets of applications that have fluctuating demand on resources.  Enabling these applications to “lease” a common set of resources only when under heavy demand can lead to huge cost saving and efficiency.

I discussed the enterprises challenges in my previous talks. The most recent one is the one I gave during the CloudSlam event:  Practical Guide to Developing Enterprise Applications on the Cloud- Online Presentation 

A good example of an enterprise customer who is already using our CCF in production is a Large Telco Service Provider that searched for ways to reduce the time to market for launching new scalable web applications. This customer was able to bring a project that previously took more than 6 months just to finish the development and testing to 4 weeks with only 90$ spent on pre-investment.

Large scale testing, performance benchmarks and demo as a service

Users that are not ready to use the cloud to run their production systems can still receive benefits from the cloud for testing and demonstration purposes. There are a few challenges that we tried to address with CCF to make this type of use simpler:

1. Making cloud testing environments identical to your local testing environment – To effectively test your application on the cloud and deploy a production version of your system locally, your application must be at a point where it behaves fairly the same in both environments.

2. Setting up a large scale environment of hundreds of nodes can be fairly complex

3. Trouble shooting failure in such an environment can be a fairly complex and lengthy process.

If your application is already running on our XAP product then porting it to the cloud and running it locally in your production environment will not require any change to your code or configuration. With CCF setting up a large scale environment of hundreds of nodes with the new CCF takes just a few minutes. We designed the CCF to run not only GigaSpaces but other external services which are essential for tests such as JMmeter, Tomcat servers etc. Users can easily configure their own initialization and command scripts when the machine is launched and define which software packages will be installed on their machines on the fly. We added the ability to capture the logs of multiple machines easily and monitor the system and middleware behavior at runtime using a zero installation environment all through the web.

GigaSpaces and our existing customers and partners have already been using CCF to run live demos and recently GigaSpaces started using CCF for our own internal large scale testing purposes. Using the cloud for demo purposes has been extremely useful to show cases that previously were hard to demonstrate such as fail-over and scalability in a real production environment. You can read more on how you can use CCF for running your own demo on the cloud in a similar way without going through all the challenges we had gone through in one of my recent posts: Demo as a Service . Similarly CCF can be ideal for running large scale Benchmarks.

Partner ecosystem -

We designed our CCF in away that would enable us to run not just GigaSpaces components but almost any application such as JEE or Spring based applications. This enables our Solution provider partners to use CCF to offer their specialized solutions and offerings for both GigaSpaces and their other software partners.

System Integrators are also looking for ways to gain a competitive advantage by offering turn key solutions at a significantly shorter time to market and lower initial investment. I tried to outline below some of the public references on these two categories:

• Solution providers

Real time production monitoring:  Dynatrace offers enhanced integration with CCF for enhanced production monitoring.  Dynatrace users are able to monitor in real time how their application is performing and trace their application bottlenecks in real time – see details on this integration here.

Model Driven Development framework - New Technology/enterprise (NT/e) - GigaSystemBuilder helps Java developers prove applications quickly - in a few days - on a local grid or in the cloud, and then rapidly move on to production development, rather than waste time on finger-trouble or implementation details – see more information here as well as a live demo on their deployment manager solution

• System integrator

In many cases customers are looking for experts that will enable them to gain the benefit of the cloud and bridge some of the skill-set gaps. AAR is one of our leading system integrators in that field who has already delivered several solutions on top of our GigaSpaces CCF for Enterprise customers. For them, using CCF was a tool to gain a competitive edge and provided their customers the ability to launch new applications in a matter of weeks as opposed to months and at a fraction of the setup (development environment, testing environment, staging environment) and hardware costs that are associated with setting up such an environment with traditional projects.

Here is a quote from one of our System Integrators in the UK that describes their experience in using CCF for the Telco Service provider that I mentioned earlier.

“We use GigaSpaces XAP and CCF to deploy a standard JEE web application on EC2. To address security concerns we kept the business logic outside of the cloud on the customer site. We exposed the business logic through a secured web services channel. We use XAP data grid for maintaining the session high availability and for reducing the latency associated with accessing the remote data center. We used the XAP integrated Jetty container for hosting our web application and through that gained the built-in self healing and auto-scaling provided through the CCF. In this way we were able to deal with potential failure or load without human intervention. From a management and monitoring perspective we were able to leverage the integrated ganglia monitoring and integrated visual representation of different metrics, like cpu/memory/network usage of the entire cluster. We used JMX to bind our custom Mbeans to the integrated ganglia UI. Overall GigaSpaces CCF gave us big boost on utilizing ec2”

I must admit that from our end this was probably the smoothest project we had been engaged with. The reason was probably related to the fact that unlike other projects we had full control over the environment. This enabled us to minimize potential miss configuration errors which is one of the major areas for complexity in setting up such a project. Another benefit was the support. When we faced an issue it was very easy to log-in directly to the machine and fix the problem on the spot without the need to go through a complex process of authorization from the IT Security and without the need to create a reproduction package and ship it over.

The customer was able to deploy a new business application quickly without putting all the initial investment upfront. In this way they gained the freedom to decide at any point if the application should continue, be canned or if they should move it entirely to their local IT.

Looking into the future

The future looks very promising as we continue to work on bringing new customers and businesses towards using CCF and the latest XAP 7.0. We are currently working on supporting the Amazon European datacenter and taking advantage of some of the new XAP 7.0 cluster administration API to enable users to provide thier own custom SLA deployment. At this point, I would like to ask for your specific feedback and wish list for our next major CCF release.

The Need for Speed - Saving Costs By Improving Application Efficiency

In times like these, improving application performance isn't a major focus for most IT organizations. The common perception is that as long as you're meeting the bare minimum demanded by your users, you're okay - anything beyond that is a luxury you can’t afford. Well, I happen to think this perception is dead wrong: these days, you just can’t afford not to invest in high performance. The reason is simple: high performance == higher utilization.

I’ll explain what I mean. If you do something that makes an application run 10 times faster (this is a typical performance boost experienced by GigaSpaces users - and by the way, XAP 7.0 will be even faster), without changing your loads or service levels, then that application will consume 90% less resources. Or in other words, you can consolidate the servers running this application at a ratio of 10:1. The amazing thing is, this isn’t instead of the server consolidation you'll get from vendors like VMware - it comes on top of and in addition to it, because it helps you cram more virtual machines and more applications onto every piece of physical hardware.

A great example of this is an eBay subsidiary, Marktplaats, which has moved its application to XAP and is now expecting to reduce their data center from a few hundred servers to only a handful - the consolidation ratio is a whopping 18:1. Marktplaats says this reduction is largely a result of the huge performance boost they experienced, which was made possible by XAPs In-Memory Data Grid and parallel processing capabilities.

XAP also makes it possible for extreme performance to thrive in unexpected places - one example is an XTP trading platform which, thanks to GigaSpaces XAP, has become SaaS-enabled, a major differentiator for the platforms makers, Orbyte Solutions. Another is our recently-announced joint solution with Mule, the open source ESB, which proves that "high performance SOA" is not an oxymoron :)

Interesting talks, and free drinks in JavaOne

Its been two years since I've last visited the JavaOne conference. This year is going to be particularly interesting as its going to be the first major Java event following the Oracle acquisition of Sun.

I will have a Technical Session on Tuesday titled: Alternative to Google Application Engine for Java™ Technology-Based Applications, where I'm going to outline the difference between the Google and Amazon approach for cloud computing and discuss how we can combine the best of the two approaches. Argyn Kuketayev and Francis de la Cruz from Primatics is going to join me  and present their experience in deploying a risk management application as a service and provide some of the technical details on how they where able to scale-out their application  on the cloud.

Daniel Templeton from Sun Microsystems will have a lab session: PetClinic in the Clouds: Scaling a Classic Enterprise Application In this Hands-on Lab, participants will take a popular Web application (the Spring PetClinic sample application) and modify it so that it can be deployed on the Amazon EC2 cloud computing infrastructure. They will be exposed to using the GigaSpaces platform as a service, in-memory data grid concepts, the OpenSpaces framework, cloud computing concepts, and persistence as a service using Sun's MySQL™ database technology.

We are also co-hosting an event Tuesday June 2 at 8PM with our partner Webtide with whom we've done a great integration for Jetty.

Among those who will attend the party will give a chance to win a free book Savvy Guide for cloud computing by Jim Liddle.

*Note: Space is going to be limited so if you want to ensure your place make sure to register on the online registration site that we set for this event.

The list below include all the sessions and labs that I hope to see – any recommendations on other interesting talks would be appreciated.

Session ID	Session Title	Session Type	Speakers and Company	Date/Time	Venue Room
TS-4605	Enterprise JavaBeans™ 3.1 (EJB™ 3.1) Technology Overview	Technical Session	Kenneth Saks, Sun Microsystems, Inc.; Marina Vatkina, Sun Microsystems, Inc.	Tuesday June 02 10:50 AM - 11:50 AM	Hall E 134
TS-4308	Architecting Robust Applications for Amazon EC2 Chris Richardson,	Technical Session	Chris Richardson Consulting	Tuesday June 02 12:10 PM - 1:10 PM	Esplanade 307-310
TS-4390	Castle in the Clouds: SaaS Enabling JavaServer™ Faces Applications	Technical Session	Lucas Jellema, AMIS	Tuesday June 02 12:10 PM - 1:10 PM	Esplanade 302
TS-3817	Google App Engine: Java™Technology in the Cloud	Technical Session	Toby Reyelts, Google; Max Ross, Google; Don Schwarz, Google	Tuesday June 02 3:20 PM - 4:20 PM	Hall E 135
TS-5454	Alternative to Google Application Engine for Java™ Technology-Based Applications	Technical Session	Nati Shalom, GigaSpaces Argyn Kuketayev Francis de la Cruz Primatics	Tuesday June 02 4:40 PM - 5:40 PM	Esplanade 302
LAB-5564BYOL	PetClinic in the Clouds: Scaling a Classic Enterprise Application	Hands On Lab	Michal Bachorik, Sun Microsystems, Inc.; Shay Hassidim, GigaSpaces; Daniel Templeton, Sun Microsystems, Inc. Wednesday June 03	Wednesday 1:35 PM - 3:15 PM	Hall E 132
TS-5214	Java™ Persistence API 2.0: What's New ?	Technical Session	Linda DeMichiel, Sun Microsystems, Inc.; Anil Gaur, Sun Microsystems, Inc.	Wednesday June 03 2:50 PM - 3:50 PM	Hall E 134
BOF-1304	Meet The App Engine (Java™) Team	BOF	Kevin Gibbs, Google; Toby Reyelts, Google; Max Ross, Google; Don Schwarz, Google	Wednesday June 03 7:45 PM - 8:35 PM	Hall E 135
PAN-5366	Cloud Computing: Show Me the Money	Panel Session	Jeff Barr, Amazon.com; Jeff Collins, Intuit; Chris Fry, Salesforce; Simon Guest, Microsoft; Gregor Hohpe, Google, Inc.; Raghavan Srinivas, Self; Lew Tucker, Sun Microsystems, Inc.	Thursday June 04 9:30 AM - 10:30 AM	Gateway 102-103
BOF-5392	Grails Integration Strategies	BOF	Dave Klein, Contegix	Thursday June 04 6:30 PM - 7:20 PM	Esplanade 307-310
TS-5307	Building Next-Generation Web Applications with the Spring 3.0 Web Stack	Technical Session	Keith Donald, SpringSource; Jeremy Grelle, SpringSource	Friday June 05 12:10 PM - 1:10 PM	Esplanade 307-310
LAB-5960	Storing Data in the Cloud	Hands On Lab	Craig Hubbard, Sun Microsystems, Inc.; Chris Kutler, Sun Microsystems, Inc.; Craig McClanahan, Sun Microsystems, Inc.	Thursday June 04 9:30 AM - 11:10 AM	Hall E 130-131

Tip* - If you want to find your own sessions I would highly recommend using the JavaOne search tool.

Seeyu next week!

GigaSpaces based solution makes it to the finalist of Cisco Developer Contest

I was very pleased to read an email from Leonardo, who was the winner of the OpenSpaces Developer Challenge (a worldwide programming contest using the Gigaspaces application server which was held last year), saying that he is now a finalist in the Cisco developer contest. Here's a bit about him and the application he submitted:

About Leonardo

Leonardo worked for several ISPs in various roles as network administrator and java programmer for IT consulting firms, and finally as software architect in high-performance Java EE based projects. He is passionate about parallel programming, distributed computing and more recently semantic web and its applications on software engineering.

Leonardo was the winner of the OpenSpaces Developer Challenge. He enjoys reading about various technologies in the field of computer science. When he is not developing code, he prefers to spend time with family and friends, walk in the park, or watch a movie.

About the application

Resource Management Platform is a proposal to develop an event based platform that leverages AXP, Services Gateway Initiative (OSGI), Jini and JavaSpaces technologies to enable deployment of IP Multimedia Subsystem (IMS) applications based on Session Initiation Protocol (SIP); more specifically, the Call Section Control Function (CSCF) components. It will have admission control mechanisms to manage Call processing.

This solution improves infrastructure manageability for large scale IMS applications. Such a platform will potentially be useful to enable deployment of high-performance, network-based SaaS (Software as a Service) or Cloud Computing solutions at the network edge by leveraging AXP.

You can find the full details about his project here.

Leonardo's project is interesting, because it shows how you can use Space Based Architecture (SBA) for implementing a scalable Telco application and offer it as SaaS application on the cloud.

Interestingly enough, I got another email the week before from Amin Abbaspour, who presented another case study illustrating how you can build a scalable SMS service using SBA, as shown in this diagram:

 

What the two projects have in common, from an architecture perspective, is that they both represent a highly scalable Event Driven design. The unique thing about Event Driven applications is that they require a combination of messaging, data and service interaction that needs to be tightly orchestrated to meet high performance/low-latency requirements without compromising on consistency, ordering (FIFO) and reliability. This combination of requirements represent one of the hardest challenges in building scalable architectures. Trying to meet this type of challenge in the traditional way by integrating messaging system for event delivery , database or simple caching (like Memcached or TC) for data and a traditional application server for business logic is going to lead to fairly complex architecture. Trying to reach linear scalability and keeping the latency low with so many moving parts is close to impossible. This is what makes SBA such a good fit. The main difference about SBA is that it recognizes there is strong dependency between messaging, data and business logic. The key is to have one shared clustering, high availability and scalability for all three components of the architecture. This makes it possible to reduce the number of moving parts and network hops associated with each business transaction, thereby increasing reliability.

On a personal level, I was very pleased to see that the software we are developing is helping people like Leonardo and Amin to build their own carrier and put themselves in a unique spot in highly competitive market.

Good luck Leonardo and Amin!

References

• Making EDA programming simple with JeeWiz
• Cisco AXP Developer contest
• What's new in GigaSpaces XAP 7.0
• GigaSpaces on the cloud
• OpenSpaces developer portal

Practical Guide to Developing Enterprise Applications on the Cloud- Online Presentation

According to a recent survey, available skill sets is one of the leading decision factors for organizations in determining which application platform to use, while scalability and availability are next. This reveals one of the main obstacles for bringing enterprise applications to the cloud: How do you take something as disruptive as cloud computing, and bring it to an enterprise environment, without forcing a complete re-write?

In my recent talk at the CloudSlam (online) Conference, I tried to summarize the challenges people face when trying to deploy enterprise applications on the cloud. Try to imagine a scenario where you have an existing JEE system, and your system is under load. At this point you would like to add more machines to accommodate that load. What will happen to your application in this case? Will your application be able to take advantage of the additional capacity? How do you know how many machines should be added to meet the load in the first place?

The answers I usually hear to these questions are fairly consistent. If you’re lucky, you’ll only require configuration changes to be at a point where you can utilize the extra resources. Knowing how many machines to add in order to meet a certain load is yet another challenge. Going through the normal capacity planning process could take weeks with the way systems are currently running.

At the same time, if you’re going to deploy our application in a static manner (reserved instances, static IP, …), it would probably make it simpler to deploy your existing application on the cloud, but it probably won’t make much economical sense. So the question I was trying to answer is how we get from the static application deployment most of us are using today, to a point where we can get end-to-end scaling and elasticity of our application from the load-balancer to the database without going through a complete re-write.

In this presentation, I tried to suggest a set of solutions to overcome the various challenges I mentioned earlier and how to apply them in a gradual manner to avoid huge initial investments and high risk. There were also some interesting questions toward the end, centered specifically around one of the hardest problems – getting the data pieces sorted out in a such a dynamic environment. Those who missed this presentation can now view it online:

Note: The voice quality isn’t that great, plus the desktop sharing on Webex didn’t give a clear indication when my desktop was actually shared, so the beginning of the talk is missing the first two or three slides. If you have questions or want to get a copy of the presentation, just shoot me an email.

During the upcoming JavaOne conference there will a hands-on lab session by Daniel Templeton from Sun Microsystems (PetClinic in the Clouds: Scaling a Classic Enterprise Application) where users can go through the steps and deploy a Pet Clinic application on the cloud. 

In addition to that lab, I'm going to have a Technical Session (Alternative to Google Application Engine for Java™ Technology-Based Applications).

For those that are not going to join the JavaOne event and want to get some hands-on experience on how the steps that i outlined in the presentation can be implemented in real application, I’d recommend looking Pet Clinic demo that is available on Amazon cloud using our cloud framework.

I’m happy to say that the number of customers that are already using this approach through our Enterprise Cloud middleware platform is growing quickly and were seeing more and more application that would traditionally considered as not "cloud compatible", being deployed on the cloud. Below are few of the public references that were mentioned just recently on Jim Liddle's blog:

• Telco application running on Amazon EC2 and GigaSpaces
• Using GigaSpaces to Trade in the EC2 Cloud

Twitter as a scalability case study – it’s the architecture, stupid!

In May of last year I wrote my first write-up on Twitter scalability, titled Twitter as a scalability case study. Back then, Twitter was still struggling with its Ruby implementation and claimed to be 10000 Percent Faster. During the past few weeks I came across the an interview entitled Twitter jilts Ruby for Scala which described some of the new development that led Twitter to switch some of its Ruby backend development to Scala. I had follow-on discussions with few colleagues ever since WRT to Scala and functional languages in general which led me to write this piece.

I must admit that whenever a scalability discussion becomes a language-choice discussion, I get irritated. It's enough to read a comment such as the one below to see that there is something is missing from the puzzle:

"Today, Payne said, most of the hip dev set codes in Ruby or PHP or Python because they're perceived as "agile" languages.. but also because the deverati grew bored with languages like Java and C++"

After I read the following note, things became a bit clearer:

"By mid-2008, one of these Ruby message queues completely crashed and developers needed two and a half hours to shove the dropped Tweets back through the system. When your game is micro-blogging, that's a lifetime. Then, in his spare time, one developer ported the code to Scala. According to Payne, the Scala queue could process the same message backlog in 20 seconds."

Later, I read Bill Venner’s excellent interview, Twitter on Scala, which i found quite insightful. But after reading the following response on the use of the Actor model in Twitter I felt even more confused:

"..we found that actors weren’t necessarily the ideal concurrency model for all parts of that system. Some parts of the concurrency model of that system are still actor based. For example, it uses a memcache library that Robey wrote, which is actor based. But for other parts we’ve just gone back to a traditional Java threading model. The engineer working on that, John Kalucki, just found it was a little bit easier to test, a bit more predictable. The nice thing was, it took minutes to switch code that was actor based over to something thread based."

Now to be clear, I have nothing against Scala or Ruby or any of the dynamic languages – quite the contrary. I believe that productivity, being agile, and enjoying writing the code are important factors when we make our language choice. At the same time we should remember that these are only some of the factors; other factors which should influence our decision are available tooling, existing skill-set, maturity, performance, etc. I also don’t care that much how Twitter makes their technology choices. What gets me worried is comments such as this one:

“the switch should stand as a lesson to cutting-edge coders everywhere”

When this type of statements gets to the press and gets backed with intelligent stories that justifies these choices, it is clear to me that many "naive" individuals will start following the same choices without understanding the full picture and history that led to the decision, thinking that they can base their decision on “proven success”. Looking at the history could evil a fairly different picture.

What I found missing in the entire discussion both here in Twitter case and in many of the other web2.0 scalabilty architecture stories is something that will indicate that there is some consistent methodology behind the  architectural choices. For example, most of the limitations that were mentioned about Ruby’s threading model, as well as its memory utilization issues, were well known before Twitter chose Ruby as their core language. Why did it take such a long time to realize that you can’t come up with scalable architecture without addressing those limitations? I could argue the exact same thing about Digg and others.

What i would recommend is to look at Brian Zimmer’s summary on Scalability Worst Practices, in particular the “Golden Hammer” point:

"The Golden Hammer refers to the old adage: if all you have is a hammer, everything looks like a nail. Many developers fall prey to the idea of using only one technology – the cost of this is having to build and maintain an infrastructure in the chosen technology which may be readily available in another technology which is more suited to the specific problem area's functionality or abstractions. Forcing a particular technology to work in ways it was not intended is sometimes counter-productive."

The lesson in our specific Twitter case is that language choice should be used as a means for implementing a given architecture and not the other way around. We may find that implementing various parts of our architecture require different technologies and languages, and that’s perfectly fine, as long it follows that order. In Twitter’s case, choosing Ruby for the front-end and Scala and Java for the heavy stuff sounds reasonable, only that I would expect them to get to this realization much sooner.

Final words

In this blog I pointed my comments specifically at Twitter, however Twitter is only an example. Many Web 2.0 sites developed their scalability architecture in a similar trial-and-error approach, which, as we can learn by their histories, tends to be very painful and costly. And yet, building a scalable Web 2.0 application such as Twitter shouldn’t be rocket science. I’m certain that if you follow the right design principles and learn from other proven scalability patterns, you can avoid a large part of the painful “trial and error” experience before coming up with the right solution. Making the right build vs buy decisions can make a lot of difference. In one of my recent posts, Designing a Scalable Twitter, I tried to suggest a methodology for coming up with scalable Twitter architecture. Like anything in life, I’m sure that there is more then one possible solution and more then one language that could be used to implement our solution. In many cases you will find that even though language and platforms could vary between different implementations, the architecture principles remain pretty much the same.

To sum that up, I would say that scalability is first and foremost an architecture choice; different languages can make the implementation of a certain architecture simpler, but language by itself doesn’t make any application more scalable. Different parts of an application may require different set of languages, and choosing the right langrage should also take into consideration existing skillset, maturity, existing development tools and best practices – and obviously, the application’s performance and scalability characteristics.

Designing a Scalable Twitter

Guy Nirpaz, Uri Cohen and Shay Banon came up with an interesting exercise as part of the recent partner training that took place at the GigaSpaces office. In this exercise, the students were asked to come up with a scalable design for Twitter, using Space-Based Architecture.

There are some interesting scalability lessons from this exercise, which are applicable to anyone looking to implement new-style real-time web applications such as the ones used for social networking.

In this post I'll  try to summarize the main patterns to put into place and considerations to make when designing such a scalable architecture.

Background:

For those of you who are not yet familiar with the service, Twitter is sort of a SMS-service meets discussion board.  You can post short messages (up to 140 characters) that can be shared with a group of subscribers that are referred to as "followers". The main difference between twitter and other messaging applications is that both SMS and Instant Messaging (IM) applications were designed primarily for one-on-one communications whereis Twitter was designed primarily for broadcast communications (publish/subscribe, or pub/sub). Another aspect that is special about Twitter is that by default anyone can follow anyone else. In other words, it was designed for open communications, not private, as were IM and SMS.

What are Twitter's scalability challenges?

1. Sending a tweet (a message on Twitter is known as a 'tweet') -– The challenge is how to handle an ever-growing volume of tweets and re-tweets and responses that can lead to a viral "message storm"

2. Reading tweets – The challenge is how to handle a large number of concurrent users that continually “listen” for tweets from users (or topics) they follow.

Designing A Scalable Twitter

Choosing the right scalability patterns

Almost every challenge in software architecture has its roots in one of the existing patterns. So the simplest course is to start by looking for those patterns, and choosing the right patterns to scale the application. Looking at many other scalable architectures, we'll begin with a partitioning pattern as the core design principle. By partitioning our Twitter-like application we'll spread the load across a cluster of servers and scale by simply adding more servers (i.e., partitions).  Another important architectural observation about Twitter is that it doesn’t fit into the classic database-centric design that most web applications do. On the flip side, it doesn’t fit well with a messaging-centric design (pub/sub) either. It is a combination of the two.

A pattern that is suitable for this type of collaborative messaging is known as a blackboard pattern.  In our design, we will use those two design patterns -- partitioning and blackboard -- as the foundation for our scalable Twitter application. With the foundation in place, let’s list the requirements and examine how these patterns can be used to scale the app.

Scalability Requirements

We'll assume a relatively extreme scaling requirement:

• Tweet Volume: 10 billion tweets per day
• Tweet Storage: 100 Gigabytes per day (with 10:1 compression)

Additional assumptions:

• Tweets are limited to 140 characters
• Tweets are immutable, i.e., there are no updates, only inserts
• Twitter limits client applications to 70 requests per hour

Now that we have the foundational patterns and clear requirements, we can design the architecture. We'll start first with the blackboard system.

Using an In-Memory Data Grid (IMDG) as a Blackboard System

The are several approaches to building a blackboard system. To maximize performance and scalability, we'll store the data in memory, thus avoiding disk I/O, which is often the main cause for contention. For years, Java has provided a model for designing blackboard systems known as JavaSpaces. More recently, distributed caching has become popular and can provide similar capabilities to those of JavaSpaces. Let's examine two popular distributed caching approaches for our blackboard system:

1. Simple read-mostly caching using memcached
2. Read/write caching, also known as an In-Memory Data Grid (IMDG)

Choosing between memcached and an IMDG

Memcached enables us to to store the data (tweets) in a distributed memory set and read it in a scalable fashion. Having said that, be aware that memcached is not transactionally-safe and is not designed for reliability (i.e., it doesn’t support fail-over and high availability). That means that if we use memcached or something similar, we will have to use a database as the back-end. Every tweet posted will have to be written to both memcached and the database in a synchronous fashion to ensure that no tweet will be lost. This approach may be good enough for scaling read access, however, for writes and updates it offers limited scalability.

Unlike memcached, which was designed for simple read-mostly caching, In-Memory Data Grids  are designed for handling a read/write scenario, and can therefore act as the system-of-record for both write and read operations. We can still use a database for long-term persistence, but because the IMDG maintains its reliability purely in memory, we can write and update the database asynchronously and avoid hitting the database bottleneck.

Todd Hoff, author of highscalability.com wrote an interesting summary that covers the different products in this space in a recent post:  Are Cloud Based Memory Architectures the Next Big Thing?

Todd provide a clear explanation of how an IMDG works (using GigaSpaces):

 

• A POJO (Plain Old Java Object) is written through a proxy using a hash-based data routing mechanism to be stored in a partition on a Processing Unit. Attributes of the object are used as a key. This is straightforward hash based partitioning like you would use with memcached.
• You are operating through GigaSpace's framework/container so they can automatically handle things like messaging, sending change events, replication, failover, master-worker pattern, map-reduce, transactions, parallel processing, parallel query processing, and write-behind to databases.
• Scaling is accomplished by dividing your objects into more partitions and assigning the partitions to Processing Unit instances which run on nodes-- a scale-out strategy. Objects are kept in RAM and the objects contain both state and behavior. A Service Grid component supports the dynamic creation and termination of Processing Units.

Back to our Twitter app: Given the scalability requirements, we will need to scale both reads and writes, and therefore, an IMDG is a more suitable approach to implementing the blackboard system.

Now let’s examine how the use of an IMDG as the blackboard system enables us to scale both sending and reading tweets. Let's start by designing the partitioned cluster.

Designing a partition architecture

One of the main considerations in designing a partition cluster of any kind is determining the partition key, such as a Customer ID in a CRM application or a Trade ID in a trading application. At first glance, it sounds like a trivial decision, but choosing the right partitioning key requires a deep understanding of the application usage patterns and data model.  In the case of Twitter, we could choose to partition the application by the data-type, the user, the tweet itself or the followers. Our first goal is selecting a key that will that will be granular enough to enable scaling the application just by adding more partitions, while making sure that we don't end up with a key that is too fine-grained -- making it sub-optimal for querying purposes.

If we use the timestamp key, for example, our application will be optimized for “inserts” (writes), however, even a simple query such “retrieve the tweets of a certain user” will force us to execute an aggregated query against all partitions. Alternatively, if we partition the data based on user-id, we'll be able to easily spread the load from different users across partitions. Retrieving the tweets of a certain user is going to be resolved in one call to a single partition. We may encounter a problem if a single user generates a significant higher load than average, however, in the case of Twitter, we can assume that this is not very likely. Partitioning by user-id is a good compromise.

Data capacity analysis

With such extreme requirements it is clear that storing all tweets in memory is going to require huge memory capacity. Very quickly this will become economically prohibitive, so we need to devise a scheme in which the IMDG acts as a buffer for most of the load on the system, and then offloads the data and queries to an underlying persistent storage.  In our Twitter example, it is fair to assume that most real-time queries (those that require fast access to the data) will be resolved in data from the last hour or 24 hours. Queries that require older data will need to hit the database for the initial call. However, subsequent access to fetch new updates should be resolved purely in-memory.

Using this approach, we'll need about 10 servers, each holding 10GB of data in memory to accommodate 24 hours of activity. If we also want to back up the data in memory, we will need double the amount of servers.

Choosing the right eviction policy

It's reasonable to assume that recent data is accessed most and older data is rarely used. To ensure that we get the maximum hit ratio on our memory front-end, let's choose a time-based eviction policy, which always holds the most recent updates in memory. When we will reach our memory capacity limit the oldest data will automatically get evicted from memory. The actual window of time in which we will be able to keep in memory is obviously dependent on the size of the cluster. With an IMDG implementation all tweets are stored in a persistent storage, which means that when tweets are evicted they are not deleted from the system.

Scaling tweet writes:

If we select user-id as the partitioning key, each user tweet will be sent to a specific partition. Multiple users may be routed to the same partition. Usually the algorithm to determine which partition fits a certain user is something like:

routing-key.hashCode() % #of partitions

In GigaSpaces, this is done by marking the routing attribute of our tweet class with an @SpaceRouting annotation.

The web front-end application will call space.write( new Tweet(..),..)  to send the tweets. This way there is nothing in our web client code that exposes the fact that the underlying implementation interacts with a cluster of partitions (spaces in GigaSpaces). Those details are abstracted within the space proxy. When the write method is called on the space proxy it parses the field that matches @SpaceRouting from our Tweet() object and uses this field value to calculate the partition it belongs to. It then uses that value to route the Tweet(..) object to the appropriate partition.

With this approach, the web application can be written in a very simple way and can interact with the entire cluster as if it was a single server.

The data from the memory partitions gets stored asynchronously into a persistent storage. The persistent storage could be a database, but it could also be other things, such as an index search engine based on Compass/Lucene.

Scaling tweet reads:

To those familiar with messaging system, at first glance Twitter looks like a classic publish subscribe application. A closer look, however, reveals that any attempt to implement Twitter with something like a JMS message queue is going to fail in achieving a scalable system. This is especially true if you consider that the system needs to maintain a durable queue for each user. That could easily lead to a scenario in which each tweet is published to thousands of subscribers and every re-tweet can potentially lead to a "message storm".

As I discuss above, the right way to think about this type of application is as a blackboard pattern, just as a blackboard (or these days, a whiteboard) is used by a group of people (followers, in the case of Twitter) to share information and collaborate. When someone writes something on the board, everyone sees it and can choose to react. Unlike messaging (take email for example), we don’t need to send separate messages to each subscriber. Instead everyone is looking at the same board. Everything is also copied from the board to paper. When the board runs out of space, we erase it. And we can always page through the paper copy to access the board history. 

In Twitter, this means that each follower that follows a group of people is basically polling for messages posted by those users from the last time he read them. To make things more tangible we can express this type of query with the following SQL syntax:

SELECT * FROM Post WHERE UserID=<id> AND PostedOn > <from date>.

The <from date> will normally be the last few minutes, if we're constantly looking for new messages.

But there's a caveat. Remember that we partitioned the application by user-id? This means that each user's tweets are stored in a separate partition. How can we read all users' posts? If we poll for each user individually, we will end up with a lot of network calls. The simplest approach would be to execute one call that looks for ALL the users we're following and look for updates (new tweets) from those users. The pattern we'll use to perform such this task is mapreduce. One way to do that with GigaSpaces is through the distributed task API:

The distributed task API is a modern version of the stored procedure. The following snippet shows what such a call would look like:

AsyncFuture<Long> future = gigaSpace.execute(new GetTweetsUpdates());
long result = future.get(); // result will be the number of primary spaces

The GetTweetsUpdates() class contains code that will be injected in each partition and will enable us to look for updates from the users we follow in a single call. Because the call runs in-process, and because the data is stored in-memory, executing such a task is extremely fast compared with the equivalent with database and stored procedure operations. Execution is aggregated to the caller implicitly. The caller can use a reducer to aggregate the results into a single result object.

Scaling the web front-end

Nothing really new here. We'll use a classic web front-end, which is comprised of a load-balancer and a cluster of web servers that act as a front end to our IMDG instances. The web application will use a single cluster-aware IMDG proxy to send new tweet posts. The IMDG proxy will be responsible for mapping the tweet with the actual partition that hosting the tweet. That logic is kept completely out of the application code. This allows us to keep our web tier clean and simple.

Keeping the web layer stateless to avoid session stickiness

One common pattern for keeping the web tier scalable is to use a Shared-Nothing Architecture, which basically means that the web tier will be stateless. This requires keeping the user session state external to the web-tier. As previously demonstrated, the IMDG can be used as high-performance, scalable data store for maintaining shared session state information. This allows us to avoid session stickiness and to scale the web tier without being locked in to a specific server throughout the entire session, in case the server is over-loaded.

For more information on how to scale the web tier, as well as other important capabilities such as self-healing and auto-scaling, see the following tutorial: Scaling Your Web Application.

Making it simple and cost-effective using cloud computing

Twitter is yet another example for a situation in which system load is highly variable and the difference between average load and peak load can be quite significant. In such cases, provisioning our system can be fairly hard and costly. This is where cloud computing and SLA-driven deployments can help us scale on demand and pay only for what we use.

Once we figured out a way to partition the application, it's going to be much simpler to package the application into self-sufficient units (referred to in GigaSpaces as processing-units) and scale the application simply by adding or removing these units on demand. You can learn more about this here

Final words

Scaling a real-time web application such as Twitter or Facebook introduces unique challenges that are are quite different from those of a "classic" database-centric application. The most profound difference is the fact that unlike with traditional sites, Twitter is a heavy read/write application, and not read-mostly. This seemingly minor difference can break most existing models for web application scalability. Using a combination of memcached + MySQL is not going to cut it for this type of application. 

The good news is that with the right patterns and set of tools, building a scalable architecture that meets such challenges isn’t that difficult.  There are already plenty of success stories that demonstrate that, such as the following example from highscalability.com: Handle 1 Billion Events Per Day Using a Memory Grid

The proposed architecture is by no means perfect and can be further optimized to meet even better performance and latency, but that will come at the cost of simplicity. I believe that the proposed architecture should get you pretty far as-is. Avoid going through more advanced optimizations until the point they are an absolute must.

References

• Twitter getting bigger - Scaling becomes more of an issue

• Memory-Based Architectures and the Cloud

• Handle 1 Billion Events Per Day Using a Memory Grid

• Are Cloud-Based Memory Architectures the Next Big Thing?

• Scaling Your Web Application

• Persistence on the Cloud – How and Why?

• Scaling twitter presentation by Guy Nirpaz

Challanges for Developing Enterprise Application on the Cloud

In the past few weeks I found myself  involved in various discussions centered around the challenges that enterprises face today when they want to deploy their application on the cloud.

These discussions were very timely as they gave me some interesting ideas for my talk next week Practical Guide for Developing Enterprise Application on the Cloud, taking place on Monday, April 20 at 11:00 am Eastern Daylight Time (GMT -04:00, New York), at the CloudSlam online conference.  

I thought that some of those discussions are worth sharing. In this post I'll try to summarize the main highlights from those discussions.

I’ll start by pointing to the following discussion thread on the cloud mailing list:  Challenges faced by developers and architects when moving to the cloud:

 Kent Langley responded by listing the following challenges in his response:

• Dealing with a lack of persistence in some cases Dealing with distributed programming models (prob. one of the most important ones imho) Having to think about the whole stack.  Not just the code.
• Caching considerations
• Messaging
• Using Memory Data Grids
• Understanding configuration management tools that might be involved Working more closely with the operations group in some case

Kevin Apte added the following comment:

There are many challenges- There is no out-of-the-box infrastructure for
hosting the typical J2EE and SOA Stack in the cloud. There is no Weblogic,
WebSphere, ALBPM, Message Bus like Tibco available in the cloud.

A development team could certainly move all of this into the cloud, but the
configuration, licensing issues etc. are all something the team would have
to solve on its own.  This is far too bleeding edge for many people.

Robert Hankel added other challenges

..   The problem of adding additional resources dynamically (e.g. more WebCache instances, or WebLogic servers) requires sophisticated distributed system management infrastructure where the entity being managed is no longer a physical or virtual box, but rather an array of boxes acting collectively as a single system..

Grig Gheorghiu posted an interesting write-up Experiences deploying a large-scale infrastructure in Amazon EC2, where he provide a very insightful summary of his lessons with deploying a large scale application on EC2.

Below are the main takeaways from Grig's summary, which I found relevant for this discussion:

1) Deploy multiple Web servers
2) Deploy multiple load balancers
3) Deploy several database servers.
4) Another way of dealing with databases is to not use them

Challenges summary?

It's easy to see that there is a common theme behind all those comments. Taking existing enterprise applications to the cloud can be very difficult simply because a) most of today's enterprise applications were built using frameworks and technologies not yet supported as first class citizen by cloud providers and b) most of those applications were not designed to take advantage of the cloud's elasticity.

Rather then pointing to my direct response to each of those challenges i thought that it would be better to provide a short summary of the main possible solutions that came through this discussion.

Does it have to be that difficult?

No. Below are two main approaches to those challenges.

- Packaging static images

The simplest approach would obviously be to package your local IT environment into images that could be easily ported to the cloud in the exact same way they run in your local IT environment; right? Well, yes, you can package anything in an image bundle and host your virtual machines in a reserved mode with fixed IP configuration. However, being able to technically do that doesn't mean that it makes sense. I would question what's the difference between this environment and any other hosting environment, and what do you expect to get by moving to such a hosted environment vs. running it in your local IT environment.

If you are going to try deploying your existing IT application on the cloud using static images, then most likely you'll end up "porting" not just the application but also the problems you were facing in your local IT environment; i.e. your application will be over-provisioned based on the peak load and you’ll end up with poorly utilized environment.

- Fully elastic application

The main driver for moving to cloud based environment in the first place was to be able to grow as you need and pay for what you use.

The question is whether you can deploy your application without changes to the application while at the same time leverage the elasticity that cloud brings.

Sounds impossible? Well, a good example that does just that is Storage. With storage you can take your existing application, run it with your local (static) disk, and then plug in a network storage and run the same application on that network device, without changing the application. In that world, instead of taking your existing local disk and virutalize it, you are taking the application and plug it to another device that has visualization built-in.

We can use the same approach as with storage; i.e. move your existing application code and run on top of a different underlying implementation that will enable you to capture the elasticity of the cloud without forcing you to re-write your entire application. If you're running in a JEE environment, it should be fairly easy. If your application has strong "ties" to back-end systems, you can use the hybrid model where your application front-end is running on the cloud, while being connected to the back-end system, using secured communication channel.

Is it really that simple?

Yes. My experience with the integration work that we had done between our Cloud Computing Framework (CCF) and EC2 showed me that getting a production-ready JEE application, including a load-balancer, self healing, auto scaling, security, database and even data grid capabilities plugged in, is actually much simpler than with any other environment I'm aware of. This is due to the built-in automation, predefined images, and the fact that I don't need to download and setup anything to get the entire system up and running. In fact, it's so simple that we decided to built our entire Demo as a Service framework around it; a framework that is used quite successfully and constantly, with customers, prospects, and now with partners.

Are there any production references?

Yes.  Read Jim Liddle's blog post and see a good example of an Enterprise JEE deployment that is already running on EC2, in production, on top of our new Cloud Computing Framework. In his post, Jim describes how this Telco operator were able to address some of the common challanges that were mentioned above such as security,flexability, cost, development complexity and lock-in as well as   high-avliability and scalability in a relatively simple manner .

What next?

In my discussion in the CloudSlam event I’ll try to provide a more detailed practical guide that demonstrating how you can take a step by step approach for porting existing JEE application to the cloud. I hope to get lots of questions and feedback during the discussion so that I can share those with you in one of my follow-up posts.

Is Cloud Computing a revolution or an evolution?

Over the past few months I occasionally got into heated debate with different colleagues regarding whether cloud computing is a revolution or an evolution. Will it change our world, or is it going to be more of the same thing with different flavors?

Well, I must admit that I didn’t made up my mind yet. One of the ways to answer this question is by looking at similar trends in other industries.

I recently came across an interesting post by Clay Shirky, Newspapers and Thinking the Unthinkable, where he discuss the impact of the internet on the newspaper industry and specifically the idea behind paying for news in similar fashion that people pay for music. I brought some of his thoughts below where he describes human behavior when a disruptive idea is brought to a well established industry:

Revolutions create a curious inversion of perception. In ordinary times, people who do no more than describe the world around them are seen as pragmatists, while those who imagine fabulous alternative futures are viewed as radicals. The last couple of decades haven’t been ordinary, however. Inside the papers, the pragmatists were the ones simply looking out the window and noticing that the real world was increasingly resembling the unthinkable scenario. These people were treated as if they were barking mad. Meanwhile the people spinning visions of popular walled gardens and enthusiastic micropayment adoption, visions unsupported by reality, were regarded not as charlatans but saviors. […]

When reality is labeled unthinkable, it creates a kind of sickness in an industry. Leadership becomes faith-based, while employees who have the temerity to suggest that what seems to be happening is in fact happening are herded into Innovation Departments, where they can be ignored en masse. This shunting aside of the realists in favor of the fabulists has different effects on different industries at different times. One of the effects on the newspapers is that many of their most passionate defenders are unable, even now, to plan for a world in which the industry they knew is visibly going away. […]

With the old economics destroyed, organizational forms perfected for industrial production have to be replaced with structures optimized for digital data. […]

That is what real revolutions are like. The old stuff gets broken faster than the new stuff is put in its place. The importance of any given experiment isn’t apparent at the moment it appears; big changes stall, small changes spread. Even the revolutionaries can’t predict what will happen. Agreements on all sides that core institutions must be protected are rendered meaningless by the very people doing the agreeing. (Luther and the Church both insisted, for years, that whatever else happened, no one was talking about a schism.) Ancient social bargains, once disrupted, can neither be mended nor quickly replaced, since any such bargain takes decades to solidify. […]

And so it is today. When someone demands to know how we are going to replace newspapers, they are really demanding to be told that we are not living through a revolution. They are demanding to be told that old systems won’t break before new systems are in place. They are demanding to be told that ancient social bargains aren’t in peril, that core institutions will be spared, that new methods of spreading information will improve previous practice rather than upending it. They are demanding to be lied to.

There are fewer and fewer people who can convincingly tell such a lie.

Now try to replace the term newspaper industry with “IT industry” and see what you get.

What are your thoughts on this matter, does cloud computing represent a revolution or an evolution?

Demo as a Service

Many of you out there who tried to run a demo online in front of a live audience are probably very familiar with the feeling when the demo is not working. Bill Gates trying to show Windows Media Center on Conan O’Brian is a good example of that :)

Having gone through this experience so many times, I know that getting a demo running successfully is not a trivial task at all.

There are many challenges involved in getting a demo right:

• Getting the demo inline with latest and greatest – every new version can break your demo. Since in most cases the demo is not going through the normal QA and testing procedures, the chance that your demo might break because of new changes get fairly high.
• Disseminating the demo among all the peers in the organization is also a challenge – if your demo is a little bit complex and requires installation of a database, a web server, etc., most likely you’ll end up with each sales rep having his own version of the demo, if at all. This is simply due to the fact that it takes time to get all those pieces to work and every update becomes a real project.
• Governing the quality of the demo – once the demo is on the hands of those sales rep you have absolutely no control over what they are running. Many will try to add their own tweaks, and without noticing end up with a broken demo or a demo that is not geared to highlight the right set of features.
• Demonstrating non-functional behavior such as scalability and availability on your own laptop is not that convincing. For example with GigaSpaces, which is a middleware product, scalability, performance and high availability are our main features. You can imagine how “impressed” I would be if you would showed me this type of feature on a single laptop.
• Now imagine what can happen in the ideal scenario, where everything worked out just fine, and the customer asked for a copy of the demo – ouch! This is where you’re going to try to find some really good excuses, because chances are that by giving the customer your demo she is going to be exposed to details you really didn’t want her to be exposed to at this stage.
• If you do get the chance to try and install the demo on the customer’s machines you might succumb to the “it’s not working on my machine” syndrome, which is often caused by environmental issues that are not so easy to spot.
• Last but not least is the collaboration challenge – in most cases demos are one of those tasks where we don’t have dedicated resources allocated to maintain and develop them. This is a classic scenario where collaboration can help to leverage the power of all the users and enable everyone to contribute to the demo in an effective manner. But this requires an initial effort to set such a collaborative environment, which the majority of us don’t spend when we set a demo. At the same time, there are only small number of users who would be willing to go through this investment themselves and contribute to such an environment.

The solution: “Demo as a Service”

The idea behind Demo as a Service is basically to apply the same principles of SaaS model to our demo.

This means that the demo would be fully accessible through the web (even if it’s a desktop application), there would be no installation required. It will be completely multi-tenant, meaning that each user can run on her own isolated environment even though she would share the same infrastructure with other users.

In this section I can refer to GigaSpaces as an example of a company that faced the obstacles I mentioned above and needed such a demo environment. I’ll share some of the principles behind the approach we eventually took to address these challenges. Assuming that others would be interested in creating a similar demo environment, we designed the system in a generic way that can be easily used by any user, even those that are not using our product, to run their own Demo as a Service without going through the same level of investment.

Here are the Demo as a Service platform design principles:

• Using cloud computing – as you can imagine, many of the things that I'll describe wouldn’t be possible if it wasn’t for cloud computing. Cloud computing enabled us to create a full production environment per demo in matter of two-three minutes, through a browser, from anywhere in the world. All you needed is an account set up and Internet access.
• No installation – only Internet access is required. I must admit that achieving this goal was a bit tricky at first, seeing as our product is not demonstrable through a web UI, and even if it was, there is not much you can see without running an application on top of it. So we had to create an environment that will enable you to deploy a demo application and watch what happens behind the scenes while the application was running. This included the ability to watch the cluster as it grows or shrink, crash machines and see how the application reacts to the event, etc. To achieve these goals, we set up the environment in such away that users can actually log into the servers running the application. We also needed to show the system view – CPU utilization, memory, etc. – of those specific machines. This meant that we needed to create system monitoring per demo and wire it on the fly so that it will monitor only the machines running that specific demo.
• Demo on demand – in our case we needed to support the option of creating a full demo environment that includes a load balancer, web containers, a Data Grid and a database, all in one click. Seeing as this happens while we’re facing users, we couldn’t afford any manual operations, and had to ensure that the entire setup would take less than five minutes. At the same time, when the demo is finished, we wanted to be able to clean the machines completely in one click. This had to be done in a way that wouldn’t conflict with other potential users running other demos at the same time.
• Fail proof – getting a reliable Internet connection is not always possible, and quite often the connection gets broken just a few minutes before the demo starts. We wanted to be able to recover easily from this type of failure. For this purpose, the web application we use to launch the demo is kept completely stateless. In addition, we keep the entire application in the cloud server repository, so we don’t need to ship any code from the desktop to the server. This enabled us to run the demo even under a low-bandwidth network. In case of connection failure, the deployment is not affected, and once the connection is re-established we are able to get the current state of the system as if nothing happened.
• Demos should be easily shared – one of the problems with demos, or any stand-alone application, is that it’s very hard to share the environment the application is running in, and therefore we found ourselves shipping different versions of the application when we wanted to collaborate or troubleshoot the application. One of the benefits of cloud computing is that I can easily share the resources I'm using over the Internet. This can also be useful if I want to interactively work with another user on the same machine, or even troubleshoot what she is doing.  All I need to do is log into the same machine and see what’s going on. There’s no need to ship any code. I can even fix issues immediately as I discover them, without needing to send long emails and package my patch for hours.
• Simple upgrade – we wanted to be able to launch a new version of our software, or the demo, very easily. We also wanted to enable users to use their current version and control when they do the switch to the new version. To achieve this, we used a shared repository model. The shared repository always keeps the latest update as well as the pervious version. The application only needs to indicate which version it wants to use, via an XML configuration file, and that’s it. In this case, we avoid the need to disseminate copies of our software with all our people or even force them to change their demos at the same time. All they need to do is to point the demo to the right version on S3, and when the demo runs it picks up that version and uses it.
• Governing the quality of the demo – now that the demo is running on a shared environment, we can ensure that everyone is using the same copy of the demo. Since we control not only the code but also the environment the demo is running in, we can eliminate any chances to the environment or human errors, which often happens when you ship a copy of the demo. If there is a fix that needs to be applied, it is applied immediately and everyone gets it. 
• Support desktop (fat-client) applications – there are cases in which the application needs to run a desktop application for its UI. In our case, part of our management tools needs to be accessed that way. For that purpose, each machine that is running can start the desktop application on the server and choose whether to launch only the application screen or the entire desktop. This is done without installing any software and without any manual setup.
• Simulate a real production environment – in order to demonstrate things like failover and scaling in a convincing way, we needed to use real machines and a real data center. This is another area where cloud computing came in pretty handy. We could easily instantiate real machines and allocate specific machines per demo and per user. This is extremely powerful, because it gives users the ability to experience the full product in a matter of minutes, without needing to download it or install it.
• Support multiple users at the same time (multi-tenancy) – multi-tenancy is a core principle in SaaS. It basically means that users can run on a shared environment while still being completely independent and isolated from each other. In our specific case, this means that:
  • All users share the same demo management portal.
  • The same user can run multiple application in complete isolation.
  • Different users can run at the same time in complete isolation.
  • Each application can be managed independently.

To achieve that goal, we had to design the management console to enable such isolation. In our case, we could use the same Amazon account to serve multiple distinct users, in other words there is no direct mapping between an Amazon user and our system user. The same user can run multiple application or different versions of the same application at the same time, and manage them separately.

Can I try it live now?

Yes. The built-in demos are available on www.gigaspaces.com/mycloud – to run the demos you don’t need to have an Amazon account, you can use a free trial user, which basically means that you’ll be running on our behalf. Note that this will only give you access to the built-in demos not your own application. A description of the available demos and how to run them is provided here.

Can I use it to deploy my own (non-GigaSpaces) demo?

Yes, you can use tihs same environment to deploy Tomcat or any other application fairly easily. Note that this option will require that you log in using your Amazon account.

An example on how to deploy your own Tomcat application is available here.

Note that this mode only provides the ability to launch new custom machines but doesn’t include all the elasticity, self-healing functionality that you would gain by deploying your web application on GigaSpaces XAP.

What’s the difference between a demo environment and a production environment?

One of the interesting thing is that there is not much difference. The difference would most likely be configuration – for example, security setup, or the size of the machines, which might be different in a demo because you would likely use low-class machines. Other than that, it’s pretty much the same environment. The beauty of that, IMO, is that you can move from demo to development to production without changing the environment.

Can I use this environment to run a POC and benchmark?

Yes. Since you get full access to the machines, you can keep them running as long as you wish, customize them and install your own software as well. This is actually the recommended way to run a POC, as it is more cost effective. You don’t spend time allocating the machines, setting up the environment and tuning the system, things that often take weeks. It is also easier to share the environment between two separate parties, e.g. the end-user and the vendor, without going through all the security processes required when you attempt to get access to the customer’s on-premise machines. In this way, you don’t need to ship code around when something is not working properly. I can log into the machine and see the exact behavior you’re seeing, and fix things immediately.

Final words

Demo as a Service is a very powerful model for overcoming many of the challenges involved in running demos and POCs. Cloud computing provides totally new capabilities, which allowed us at GigaSpaces to create a full demo environment for our product, at the click of a button. Having said that, based on our experience, setting up a full demo environment can take a while and would require investment. The mycloud environment was designed to get you the benefit of Demo as a Service while saving you the effort required to implement one. It is still in its early stages but is already being used extensively by our sales team and our partners. I’m eager to hear more feedback from users and suggestions on how to can make it better.