Bringing Data Awareness to the Grid (and Amazon EC2)

For those who aren't familiar with the concept of Grid Computing, a brief primer:

Grid is a well known paradigm designed to reduce the time it takes to perform a batch job through distributed parallelization.

How does it work?

Most grid solutions are built out of three main components:

1. Compute Agents - a lightweight process that makes a compute resource grid-aware; i.e., available to the grid.
2. Resource Manager - collects and monitors information about availability of compute resources, as well as maintaining the liveliness of those resources.
3. Scheduler - this is where the heavyweight work is done. The scheduler is responsible for matching between the task specification and the available resources.

Different solutions in the market (such as DataSynapse, GridGain, Grid Systems, and Platform Computing) refer to those three by different names, but in general the principles are pretty much the same.

Using a Compute Grid allows us to break a batch job into smaller units of work, a.k.a  tasks. At this point we can submit the job to the scheduler, specifying the requirements and specifications (Service Level Agreements) required for running the job. Requirements can be specified as a type of OS/HW pre-requisites, etc. The scheduler then takes those requirements and executes the job on the available compute resources that match those requirements. Compute farms can be as large as 10Ks of CPUs.

This approach was used with great success in the past few years, and today most financial institutions have a compute backbone as a core piece of their IT infrastructure.

Utilizing Compute Grids for a Wider Range of Applications

It is now time to take advantage of the success of Compute Grids and apply it to a wider range of applications within the organization, including to those that are considered very high-end, such as front-office applications in capital markets (trading, etc.). A key for enabling this type of applications on the Grid is solving the data bottleneck. This is where In-Memory Data Grids come to the rescue.

Victoria Livschitz from Grid Dynamics gave a presentation in Spain last week During an HP event in which she discussed the various options for integrating the two complementing technologies. Her scenarios ranged from a relatively trivial scenario in which the two solutions “live” together, side by side, in the same environment with very little integration between them; and ended with a complete data-aware scheduling, in which the compute grid becomes data-aware and is using this awareness to optimize the way jobs are routed and executed. Livschitz describes a case study for creating inter-day P&L (Profit and Loss) reporting for a trading application using the GigaSpaces In-Memory Data Grid and DataSynapse Compute Grid. In that scenario, Grid Dynamics managed to reduce the application start-up time from 40 min to 15 min and access time from couple of minutes to seconds, by putting the database behind an In-Memory Data Grid. This enabled the move from post trade reporting (over night) to inter-day reporting. View Livschitz' presentation (pdf).

Compute Grids for the Masses

Due to the large amount of required resources, Grids were so far mostly used by large organizations, such as major financial or manufacturing organizations. However, the introduction of Amazon Elastic Compute Cloud (Amazon EC2) brings the grid to the masses, and with the recent upgrades made to this platform, it is becoming applicable to both low-end and high-end applications. In fact, at GigaSpaces, we're developing an integration with Amazon EC2 that would make it data-aware and therefore will meet the tough requirements of those high-end applications. As Geva Perry wrote in his blog post Scaling Stateful Applications on Amazon EC2:

...we've made available a public AMI (Amazon Machine Image) for the GigaSpaces XAP platform. We're still working on improvements and optimizations, but it is ready for people to start playing around with.

• You can find the GigaSpaces AMI here.
• General description in the Amazon Web Services Solution Catalog here.
• Detailed paper with set up instructions and code examples here (PDF).

We're already working with a couple of beta customers on EC2, including some folks who are building a Web 2.0 social networking type sites.

Stay tuned for updates on this topic!