Nati Shalom's Blog

Saving cost using Application/Middleware virtualization

Earlier this week, I gave a joint webinar with James Liddle, where we outlined
practical guidelines for saving costs using middleware and application-level Virtualization:

• Saving the cost of peak/static provisioning using on-demand scaling
• Saving the downtime cost
• Saving costs through outsourcing part of our application and operations to the cloud
• Saving costs using application level optimization (doing more with less)
• Saving costs using platform consolidation to reduce the number of software components as well as utilize OpenSource and more commodity Software packages

Towards the end (Slide 20), Jim Liddle presented real life case studies from the iPhone launch in the UK and how some of hose principles have been applied to enable a successful launch in the UK.

Additionally, Jim went through some of the motivations and case studies that led different Telco, Online Gaming and Start-up companies to utilize our cloud, offering to gain better cost effectiveness.

For those who did not have the chance to participate in the webinar, we uploaded a recorded version of that presentation for you to view.

Savig cost using application level virtualization

View more presentations or upload your own. (tags: computingvirtualization cloud)

I would like to point out a few specific items from this presentation:

Beyond Server Side Consolidation (Slide 5)

Server-Side-Consolidation (SSC) played an important role in bringing the concept of virtualization to mass adoption. It forced organization to be able to map their application into concrete packages and look at machines as a logical entity rather then just a physical entity. Server-Side-Consolidation also brought a relatively simple model for cost saving: Instead of running applications on dedicated HW you can consolidate them into one machine. By doing so, you can reduce the number of servers and save the hardware and the operational costs associated with that.

Having said that, SSC is only one *very narrow* aspect of virtualization that unfortunately became too coupled with Virtualization.

The next step is obviously to move from SSC to application and middleware level Virtualization. Application Virtualization refers to the opposite scenario. Instead of putting multiple applications on the same hardware, we are taking a given application and spreading it on a pool of machines. This holds significant potential for making applications more efficient. For example, just think of the saving potential gained by moving applications from static peak-load provisioning to on-demand provisioning. In addition to that, we can utilize commodity hardware resources to get the power of high end machines.

Practical steps ( Slide 19)

One of the main concerns that most people have WRT to application level virtualization is the effort associated with applying those principles.

The table below captures the value vs effort that we can receive with each part of our middleware and application level virtualization.
 

As you can see, there are steps that require non or very little changes to our application:
For example, taking an existing web application that requires 10 machines to meet certain peak load, however on average it needs only 3 machines.  Normally we would statically provision 10 machines to meet the peak load which means that on average those machines would be poorly utilized. .  Instead, we can provision our web container on-demand and use Pay-Per-Use model to pay only what is consumed. This will enable us to save roughly 7 machines (10 - 3 ) and wouldn't require any changes to our code. (You can learn how you can do that with GigaSpaces here)

If we have a computational business logic or even a rendering application (a good example of that is  Slideshare or Yutube). Those type of applications tend to deal with fluctuating loads. So if on average we consume 10 machines and for peak we need up to 90 machines - you will be able to save ~80 machines! by provisioning the computational/rendering machines on-demand. (See here on how to use GigaSpaces for MapReduce computation and here to learn on how to use the Actor model)  

On the messaging and data side, we can reduce the amount of machines and the cost of scaling by partioning our messaging or data thus enabling linear scaling of those layers. In addition to that utilizing memory resources instead of files provide a significant boosst in performance. The combination of the two enables us to utilize commodity software and hardware resources instead of high-end resources - for example we don't need to rely on high-end databases, we can simply use MySQL.

Final Words

In this presentation we tried to gather most of the knowledge based on our past experience. Most of these lessons are generic and not necessarily specific to GigaSpaces.

I also tried to provide some practical implementation guidelines (Slide 19):

• Avoid radical change, enabling a gradual process
• Choose an architecture supporting linear scalability
• Minimize vendor lock-in
• • Enable application portability and freedom of choice of:
  • • Cloud provider, Web container, Programming language, Database
  • Minimize API lock in:
  • • Use of standards
    • API Abstractions – when standards are not available
• Future proof your application
• • Don’t make decisions today, but be ready to make one without major effort
  • Avoid long-term commitment – choose the right licensing model

Real life case studies:

•

I would like to encourage you again to listen to the case studies (Slide 20) and learn how others  apply some of those principles in their local-IT and Cloud.

Additional references:

• New quick-start guide shows how you can easily scale your on-demand scaling to your EXISTING web application.
• New whitepaper: Survival Guide for IT During the Economic Meltdown
  

GigaSpaces as Alternative to GoogleAppEngine for the Enterprise

I Just came across an interesting post by Josh Heitzman who writes about his negative experience with Google App Engine, which led him to examine a list of Alternatives to Google App Engine. He points out GigaSpaces XAP as one of them:

One particularly interesting EC2 third party provider is GigaSpaces with their XAP platform that provides in memory transactions backed up to a database. The in memory transactions appear to scale linearly across machines thus providing a distributed in-memory datastore that gets backed up to persistent storage. A lot of the docs reference Java, but the page returned by the aforementioned link states “…deploy applications that use Java, .Net, C++, or even scripting languages…” so after a cursory investigation it is not clear what aspects of their platform is only accessible via Java and which aspects are generally accessible. Bears more investigation.

[See my comment to the post relating to the question about .Net and C++ support]

Josh's post raises the question of what is Platform-as-a-Service (PaaS)?

Platform-as-a-Service is a term used to describe a new set of development platforms that are typically accessible through the web. These platforms enable you to develop new applications easily, without the need to install any software or set up a development and deployment environment. A good example of this is Force.com from Scalesforce.com. Other SaaS providers have similar platforms. It seems that the common motivation behind this trend is to enable the SaaS provider a way to expose their internal framework to other partners and users and build an eco-system around their SaaS product. This led to the emergence of dedicated, proprietary platforms. Google App Engine is a similar effort from Google initiative to expose some of their own platform to external users.

These platforms, as Josh experienced, were not designed as a general purpose enterprise platform, and therefore, it is not surprising that they lack many of the elements that you would normally expect from enterprise middleware, such as transaction support, security and standard APIs.

Unlike such Internet based platforms, GigaSpaces XAP was primarily designed as an enterprise middleware platform. It is used in the most demanding mission-critical applications that require extreme scalability and low-latency. During the past year we have extended our middleware platform to the Internet cloud, starting with tight integration with Amazon EC2 and followed with partnerships and integration with leading players in the market, including GoGrid, Joyent, RightScale, CohesiveFTand others. We recently launched our cloud framework in private beta. It enables building enterprise applications on GigaSpaces XAP via the internet. In this way, you can run an application on a hosted GigaSpaces environment, without even downloading the GigaSpaces software.

What makes GigaSpaces XAP an alternative to Google App Engine for the enterprise?

• Support for existing enterprise applications:

One of my previous posts discussed: Google App Engine - what about existing applications?

In this post, I want to reiterate this point, which goes to the heart of one of the main differences between GigaSpaces XAP and most Internet based PaaS, including Google App Engine and Force.com. Many enterprise applications are already built in Java, .Net or C++. To support enterprise applications, you first need support for these core languages. GigaSpaces XAP not only supports these languages, but also enables efficient interoperability among them.

• No vendor lock-in: 

While I would argue that lock-in is unavoidable at some level, and that every platform imposes some lock-in, I'd also argue that it is important to examine the nature of the lock-in, and how easy it is to migrate from one platform to another. With XAP, we invested heavily in making lock-in minimal through abstraction, aspects, support for standard APIs and more. As of 6.6, we users can take existing web applications and deploy them on our platform without touching the application code. You can read more about this in: Can scaling be made seamless?
To make things easy, we published a migration guide that shows step-by-step how you can take existing transactional JEE applications and deploy them on our platform (locally or on the public cloud). We measured the performance and scalability gain you get by running JEE applications on GigaSpaces XAP versus traditional JEE application servers. We ran the exact same application code on both platforms and measured at least 5 times better scalability efficiency and performance increased as outlined here. We also published a new "pet clinic" demo that comes with source code, configuration and documentation, and can be used as a reference guide for running standard JEE application with zero or with minimal code changes. This reference implementation is available here.

• Designed to support both local enterprise clouds and Internet clouds:

Although GigaSpaces can now run entirely on a public cloud, such as EC2, it is clear that many enterprises are not ready to run on public clouds, but would rather run their apps on an on-premise, private cloud. Supporting this requires existing development tools used to develop enterprise applications. XAP enables use of common development frameworks (e.g., Eclipse, Maven, Ant, in Java; and Visual Studio in .Net). You can write, test and debug your application locally and then deploy it on the cloud for testing or for production. You can decide at any point where you want to deploy your application, whether on a public Internet cloud, or on a private cloud in the corporate data center. You can also create a hybrid model that involves both a public and private cloud simulataneously.

•  Enterprise-grade reliability and scalability:

Most Internet-based PaaS impose a radical shift in the way applications are built, and specifically on scalability and reliability. Many of them leave you to deal with failure scenarios on your own, or alternatively, force you to accept the fact that you may lose data if you want to achieve scalability. They also require that you re-write your application if you want to make it scalable.
Many of the assumptions the platforms operate under are not acceptable to enterprise-grade applications. GigaSpaces XAP was designed to meet the most demanding requirements for maintaining both scalability and 24/7 high-availability without losing any data and without compromising scalability or performance.

This is only a partial list of differences, but I think that it makes clear how GigaSpaces is different than most Internet-based PaaS offerings, including Google App Engine.

You can read more about our cloud offering on gigaspaces.com/cloud. If you are interested, try out our new Cloud Framework and see for yourself how easy it is to set up a production-ready cluster with load- balancing and scalability within minutes.

GigaSpaces XAP 6.5/6.6 new releases

GigaSpaces 6.5 was released at the end of June, and we are now working on the 6.6 release, with the first milestone already publicly available. These are major milestones in a series of upcoming releases all aimed at strengthening our proposition as a Scale-Out App Server. Our main goal is to significantly simplify the process of achieving scalability, including scaling an EXISTING application within days and without enforcing a complete re-architecture.

I refer to this as our "Seamless Scaling" or "Simple Scaling" initiative. You can read some of the rationale behind this initiative in my previous post Can scalability be made seamless. This is a very ambitious goal, and by all means, we are not finished. In addition to the tremendous enhancements already put in place, we have a long-term roadmap that covers many aspects of the product.

The efforts we have undertaken (as well as those on our roadmap) involve enhancements to our development frameworks in Java, .Net and C++, mostly around the abstraction layer, including supporting standard APIs that enable us to inject many of our Data Grid and event-driven capabilities through annotations and configuration, with zero or minimal changes to application code.

They also involve increasing robustness, making large-scale deployments simpler to deploy and manage. Other efforts include extensive integration with popular frameworks, such as Spring and Mule, and recently we also added Web framework integration with Jetty. All this is designed to make the end-to-end scaling experience extremely simple and native. Judging by recent feedback we received, some of it publicly referenceable, it looks like we're making great strides in achieving our goal. I particularly liked the following quote by one of our existing customers Monte Paschi Group, who built a new pricing system with GigaSpaces. Their full case study is available  here. I chose the following quote from this study as it highlights some of the benefits that we don't always emphasize enough - development simplicity:

The development team is happy, too,  since the architecture has been
greatly simplified compared to the multi-layered application server system.
"We're not a software company, we're a financial company," Santini
explains. "We didn't have weeks or months to study the technology. Our
main goal was to use it to achieve our goals. GigaSpaces XAP allowed us
to do that right out of the box.

You can read the full details of the 6.5 release here. For convenience, we grouped separately the long list of features into Java,.Net and C++ categories, and provided detailed descriptions that outline the rationale and the value behind each feature.

In this post I'll try to highlight some of the important features and provide insight into our future plan.

Seamless scaling using the Service Virtualization Framework

At the application layer the most notable feature is the Service Virtualization Framework. The Service Virtualization Framework (SVF) can be seen as a major enhancement to Session Beans in EJB3 and Spring Remoting. This  framework enables you to write your business logic as a POJO and deploy the services across a cluster of machines, while providing a single client proxy that virtualizes all those instances as if they were a single server. For more details I recommend reading a new white paper covering the concept behind this framework and how it can help you simply build scalable, high-performance SOA and event-driven applications. The white paper is available here

Seamless scaling of popular development frameworks

We enhanced and expanded our integration with popular development frameworks. The purpose of the integration is to provide end-to-end seamless scaling to such frameworks in a way that doesn't require changes to the application. A good example is our Mule-ESB support. Mule users can take existing Mule 2.0 applications and significantly improve performance and scalability by plugging in the GigaSpaces runtime into the Mule Framework. The good news is that the wiring happens outside of your application code at a couple of levels:

1. Connector level – leveraging our messaging layer as the transport for Mule
2. Clustering level – taking advantage of our clustering capabilities, enabling the internal Mule data structure to span across multiple machines for scalability and high-availability

This integration is provided as part of our open source framework, OpenSpaces. We hope and anticipate that these integrations will be used as a reference by other frameworks looking for ways to provide similar levels of scalability and reliability. A good example for that already happening is the work David Greco performed by integrating the Camel open source ESB with GigaSpaces.

With 6.6 we also added out-of-the-box integration to Jetty. This was done in collaboration with the Webtide team (the company behind Jetty), who have given us excellent support throughout the process. What i like about this integration is that it enables taking an EXISTING web application packaged as a WAR and dynamically scaling it across a pool of machines. With this approach, you also get session-replication injected to your existing application without touching your code or WAR package. If you're willing to make slight configuration changes, you can get caching reference injected into your application. There is a new example that shows what it takes to scale an existing web application. The example uses the Spring Pet Clinic application and deploys it on a GigaSpaces cluster. The full example is available here.

Removing the language barrier

For decades language have been treated almost as a religion by developers. As an ex-CORBA guy, I know how much language interoperability is painful to deal with, and often requires compromises on functionality, performance and complexity. At GigaSpaces, we realized that there is no reason that different languages shouldn't be treated simply as forms of writing business logic. They each generate different values. For example, .Net provides better integration with Windows applications. C++ provides better performance in certain areas and provides low level APIs for integration with many third-party libraries.

Persistence: 6.5 has some major enhancements for implementing our Persistence-as-a-Service model in .Net through support of nHibernate. This enables .Net applications to integrate with existing databases and have their own database mapping layer with a native .Net API. This comes along with quite extensive Perforamance improvements. You can see some of the results here for .Net, and here for C++.

One of our goals for 6.5 was to bring the same level of scalability and simplicity we provide for Java to .Net and to C++, without compromising performance. Unlike some of the alternatives in the market, we don't just provide remote access to our Java runtime, but provide complete application server capabilities in these two languages -- as well as complete interoperability among all three. Java, C++ and .Net services and clients can run and share the same process and leverage that to remove the network call overhead often required when a call crosses language boundaries. An immediate benefit is that you're able to run your C++ and .Net business logic where the data is. Furthermore, you can now leverage our existing SLA-driven deployment to automate the deployment of Java and .Net applications. This means that instead of running each server process manually, you have a single deployment command that will make sure that your serves are running on the appropriate machines, that your backups are running on different hosts from your primaries, that if one machine goes down a new instance will take over immediately, or if one is not available, as soon as it becomes available -- all that without any human intervention!

Dynamic language support

The Java framework guys realized the need to support dynamic language as part of Java, making the JVM a common platform for running various languages. GigaSpaces XAP 6.5 leverages this, and provides enhanced support for Groovy, JRuby and JavaScript.

Dynamic language support enables writing business logic in Groovy/Jruby/JavaScript and executing it on the GigaSpaces cluster. One of the common use cases for this capability is to provide an elegant alternative to Stored Procedure. this means you can write business logic in Groovy, for example, that will be executed directly on the data grid nodes. With this, you can write your own custom data-queries and aggregation functions, and execute them where the data is. Beyond the performance benefit that you gain out of running the logic collocated with the data, you gain the benefit of using dynamic languages, i.e., you can add new functions on the fly without the need to deal with class-versions and class-loading issues and without the need to bring the data down whenever you do that. In this way you can add new functions while the system is running and continues to serve other applications.
This feature leverages the SVF mentioned above. This means that you can choose to run these dynamic procedures synchronously, asynchronously, in parallel, etc. Now Isn't that cool?

Click here for code snippets and detailed descriptions of this feature.

Data awareness everywhere
Throughout all of our development efforts, we are making sure data-awareness is maintained across the entire stack. Data-awareness means that invoking a method on the new Service Virtualization Framework can be routed to a particular service instance based on the data associated with that service instance. It also means that when you send a message through our JMS implementation, you we will be able to route it to the JMS partition that manages the relevant data. Unlike alternative solutions, this is native to our environment, meaning that there is no need for external integration and complexity to achieve this behavior.

Click here to view a code snippet and detailed description of how routing is handled in the Service Virtualiztion Framework.

Performance, Performance and more Performance...
Improving performance remains a constant goal for all of our releases. As the product matures, finding places in the product where performance can be further optimized is getting harder, and I therefore am always surprised when one of the developers comes up with some creative idea around performance.

In this release we improved performance on several fronts -- including Java, .Net and C++ -- which involved significant optimizations of object serilization and multi-core scalability. For the latter we are working with Azul, and making it part of our testing environment, as well as other multi-core systems such as Sun Niagra. You can see some of the figures and details here, here (Comparison with previous release of .Net) and here(C++).

We conducted detailed comparisons of latency and throughput of a "classic" transactional application based on the standard JEE model (Using JBoss ,JMS, Spring, Hibernate) with the same application but using GigaSpaces as the messaging and data-layer -- and eventually replaced the entire JEE stack with a GigaSpaces + Spring stack. It is important to note that throughout this process, the business logic code remained untouched. The initial results of this tests can be found here:

You can find the details of the code that was used in this test and the migration steps in a new whitepaper that is now available on our site here.  Uri Cohen wrote up in his blog (The Space as a Messaging Backbone) some of the interesting findings from this analysis that showed the difference between end-to-end measurment and point optimization and why in some cases putting a distributed cache in front of a database is not going to be enough.

What's next:

For obvious reasons I can't expose our entire roadmap as of yet. What I can say for sure is that we're going to continue improving the level of seamless and simple scalabaility provided by our platform.

We view the partnership and integration with other frameworks as strategic, and we're going to continue with that effort. One of the frameworks we are planning on working on is GlassFish.

We already announced our first cloud offering, designed to run on Amazon EC2, and including partnerships and integration with RightScale and Cohesive FT. If I'm not mistaken, this was the first Java application server available in a pay-per-use model, designed to meet the needs of enterprises and ISVs that want to offer their applications on the cloud, including as Software-as-a-Service. We're going to put in a lof of effort into making cloud deployments simpler, enabling our customers to use it on their local virtualized environments (private clouds) and on public clouds (Amazon EC2, GoGrid, FlexiScale, AppNexus and others), or even a combination of the two, without changing their applications. We're now working on a new version that enables provisioning a cluster of machines, deploying the application on said cluster and opening up an adminstrative console for the cluster -- all with a single click. This is already working in an internal beta. We're planning to provide a preview release by next month. With the availability of Windows-based clouds, we will be providing our .Net application platform as a cloud offering as well.

On the API and Standards front, we recently joined the OSGI alliance, where we expect to play an active role.  We are also looking into ways through which we can strengthen our compliance with some of the latest standards on the JEE stack, such as EJB 3.0 and JPA. The challenge is not just basic API mapping, but how to do it in a way that doesn't break our scale-out architecture and doesn’t create complexity. Unfortunately, previous versions of the EJB spec weren't a good fit. EJB 3.0 looks much more promising.

On the .Net front, we're going to continue with our performance optimization project. We're also working on making our .Net offering fit natively within a .Net development environment by providing better development and installation packages that fit better with the .Net spirit. We are also looking into ways to simplify the testing and debugging process. For pure .Net users we will make the .Net version available as a standalone package at a reduced price (details will follow).

On the C++ front, we're going to provide our customers with an open source version of our C++ binding and a complete package that will enable them to compile and build our C++ with their own set of dependencies, libraries and compiler versions and flags. This will also allow using the current C++ framework as a broad integration framework for third-party tools and languages.

There's much more than I could cover in this post. I tried to put together what I thought are the highlights of the release. As it's impractical to cover such a wide spectrum of topics in a single post, we started a process in which different people from our R&D and field engineering teams will post on specific aspects of the product and best-practices for using GigaSpaces in existing web, financial, online gaming and other applications.

Be part of our next release:
As we are now making the decisions of what to include in our 7.0 release, it would be nice to hear your feedback and specific requests for enhancements or new features. You can either send me a direct email or send it to PM at GigaSpaces.com Alternatively, if you think that you have a good idea that other users might be interested in, you can implement it on our community site – OpenSpaces.org.

The new GigaSpaces XAP 6.5 is available for download here.

Can scaling be made seamless?

Putting together the two words "seamless scaling" in front of a technical audience is a very dangerous thing to do. The technically savvy folks are walking around with plenty of scars from previous attempts to scale their system - enough to know that there "scaling" and "seamless" couldn't be further apart. But nevertheless, in this post I'm going to take the risk and do just that :)

Basically what I'm going to try and argue is that while scaling can't be made seamless across the board, there are different techniques to make scaling seamless in certain scenarios, or at least very close to seamless. I will use GigaSpaces as an example of how to achieve seamless migration of existing JEE applications into a scale-out model, with zero or minimal change to the code. I'll also outline our general principles, which I believe are applicable to any application seeking seamless scaling.

The seamless scaling dogma

There has been a lot of discussion over the past year about different patterns of scalability. I devoted quite a few of my posts on this topic. Most of them centered around architecture - how we can use partitioning to avoid a data bottleneck, how we can use in-memory implementations to get better performance and concurrency compared to implementations based on the file-system, and how we can use an asynchronous event-driven architecture as a better way to scale our business logic.

Randy Shoup outlined these principles nicely in his infoQ article, Scalability Best Practices: Lessons from eBay. The dogma behind all these discussions and panels was that scaling requires a very rare set of skills, which average developers don't have, and that's why we're still seeing plenty of online system failure. The most recent was the iPhone launch failure.

Does scaling really have to be complex?

Well, if you look at Network Attached Storage as an example, you'll see there are alternatives to the traditional dogma around scaling. With storage systems, we don't really think of scaling that much. More so - our applications don't really need to be aware of the fact that they run over a local disk or a network -attached device. We can scale by adding disks, even hot-swapping them in some cases, even while our application is still running.

Now imagine what the world would look like if it wasn't that simple. If our application would need to be aware of what's behind the scenes of these storage devices and would have to be re-written to deal with these scaling issues. It's not that hard to imagine, is it? Most likely we would still have been talking about storage-related system failure as a result of bad architecture and implementations issues. But we don't have anything to talk about, because storage gave us a level of abstraction that enabled almost everyone, regardless of their skill-set, to deal with scaling without being an expert at it, or really even thinking about it much at all.

Can we learn any lessons from NAS about our ability to achieve seamless scalability?

Let's see what were the conditions that made seamless scaling with storage possible:

• Well-defined interface (or abstraction)
• Interface that fits the share-nothing approach to make it suitable for scaling
• Simple interface
• Widely-used interface

Now if we examine these principles as they apply to other layers of the application stack, we'll get a decent answer as to why we haven't been able to apply the same level of seamless scaling - which storage already provides - to these other layers.

In the data layer, the most commonly -used interface is SQL. SQL fit well with criteria (1) and (4) criteria but doesn't meet (2) and (3). HashTable fit well with (2) and (3) but unfortunately is less commonly used in distributed systems. JavaSpaces, like HashTable, fits (2) and (3) but is even less commonly used then HashTable. In the messaging tier, JMS fits well with (1), (3) and (4) but doesn't lend itself well to (2), and so on. And these are the cases where there is a well-defined standard. Unfortunately, in other layers of our applications it's even harder to find a well-defined standard that fits to all of these criteria.

To overcome this complexity, there have been other attempts to use the JVM bytecode as a lowest common denominator and introduce seamless scaling not at the middleware API level, but on the JVM level using bytecode manipulation. This seems like an elegant solution to the problem, however most of the existing distributed systems were not written as a standalone Java applications that get distributed by some sort of magic, so it fails mainly on the 4th criteria - it fits mainly to new applications that were designed with certain assumptions in mind about how the standalone Java code would behave in a distributed environment.

Now to the point - can we scale seamlessly?

Those who expect a simple yes-or-no answer to this question are going to be disappointed - there is no clear answer , because it depends on the specific application scenario, the way the application was written and the maturity of various standards around these applications.

In general I would say that Java-framework-based applications are in better condition then applications based on other frameworks, due to the maturity of the standards and the advanced layer of abstractions that are now available as part of framework such as Spring and Mule.

Seamless scaling at the application layer would most likely mean the ability to plug-in different underlying scalable implementations at the middleware layer (data, messaging, business-logic, presentation). The use of abstraction layers such as IOC in Spring/Mule and the new EJB3 abstraction gives more freedom to plug in different implementations that don't necessarily conform to the exact same standard API. That means that your code can remain intact when you plug in a different messaging implementation, for example, whether it is a JMS implementation, a space-based messaging, or remoting.

Some cases are going to be easier then others. For example, taking a SessionBean and scaling it by having multiple instances of that service running over a pool of machines, while viewing them all as if they where a single server, can be done through configuration changes only. We can do pretty much the same thing to the messaging layer, where we will have a virtual queue and topic rather then a centralized server.

On the data layer things are more tricky, as most of the commonly-used standards in this area don't fit criteria (2) very well. If our data model is built with a complex object graph, or if our queries depends on complex joins, then we're not going to be able to scale it out without changes to the code or to the domain model. But even in these more difficult cases, it's possible to minimize the scope of change by using the DAO pattern, declarative transactions and annotations as a mapping layer on top of the domain model. This means that even if the change can't be completely seamless, it will nevertheless be quite simple to achieve.

Learning from the GigaSpaces experience

At this point I'd like to use our specific experience at GigaSpaces to describe the methods we used to enable seamless scaling:

• Use standard APIs, but only when it makes sense. For years we chose not to implement large parts of the JEE standard, such as EJB and Entity beans, because they didn't fit the scale-out environment and were too bounded to database. What I'm trying to say is that implementing a standard API is not always going to make the transition to scale-out model seamless, so you should be careful which standard you pick.

• Leverage existing abstractions to plug in different implementations that are based on other APIs or technologies than the one originally used. We use this principle quite extensively in our OpenSpaces framework, to map our own transaction handlers, Remoting abstraction, to enable seamless scaling of SessionBeans , etc.

• Use annotations for mapping between different models.

• Use aspects to add new behavior when it makes sense. We use aspects in several cases such as filters/remoting aspects and security aspects. We will probably be using aspects more to address a more advanced level of serialization.

• Apply more tightly coupled integration to specific products/frameworks  A good example for that is our Spring, Mule and upcoming web tier integration. This sort of integration enables an end-to-end seamless scaling story that makes the user experience significantly better. On the .Net side our integration with Office and Excel enables something equivalent.

• Use open source as a tool to open up the framework for extensions and other integration work. This is something that we introduced quite recently through our new OpenSpaces.org community site and found it to be a useful tool with many extensions already available. GigaSpaces users implemented their own extensions and made them available through the community site. The most recent one has been Camel integration.

Real life examples

Of course, this isn't just a theoretical discussion - we've been attempting to achieve this level of seamless scaling in practice since we introduced our middleware virtualization stack, which was our first attempt to address scaling of existing applications and not just new applications.

We have been involved in numerous scenarios of scaling out existing applications. An interesting example is detailed in Mickey's recent blog post, in which he describes in more detail how he was able to scale-out a JBoss/Oracle RAC-based application. Mickey provides a good description with code snippets that show the before and after effects, both in terms of code changes and obviously scaling and performance. You can find the details of that experience here. The bottom line of this case study is the fact that he was able to get that application from 15tx/sec to 1500tx/sec in less then 4 days! For me, measuring the time it takes to move your EXISTING application and see the immediate results is the ultimate measure. You have to agree that if the transition to a scale-out model wasn't seamless, it wouldn't have been possible to do in such a short time, and more importantly, without ripping and replacing the entire application. In Mickey's case, we started with decoupling of the database to get the initial scaling, and replaced the other layers incrementally.

Summary

Storage taught us the lesson of seamless scaling. Seamless scaling can be achieved on other layers of our application as well, using a combination of Standard APIs, Abstractions, Aspects and tailored integration. In most cases, seamless scaling would mean no changes to our application code but would require changes to configuration and packaging. Not all layers can make a fully seamless transition. But in those more difficult cases, we can use the same principles to significantly minimize the changes required for scaling.

In this post i wanted to share some of our GigaSpaces experience in that area as i believe many of the lessons and principles are pretty generic and can be applied to any project/product. At this point it is also important to note that this is not a one-off proposition. It's a continuous effort and requires a long-term roadmap and commitment. We've been struggling with this for years and applied every possible method to achieve this goal. Some required significant re-factoring of our entire infrtustructure. The lastest one has been the addition of our OpenSpaces framework as an open source development framework based on Spring. With this change, we can easily support more APIs and frameworks, as well as build an entire ecosystem around it that will enable others to apply the same model to even more frameworks and applications very easily.

You may wonder why we, as a commercial company, would want to do this - after all it also means that GigaSpaces can be replaced much more easily. Well, the reason is fairly simple - we believe that our success and adoption will be much higher if we can get to the point where scaling any application through GigaSpaces won't require any changes to code. It took few years and an intensive effort to get a point were I can feel comfertable to use the two words "Seamless Scaling". Now we're starting to see the fruits of that effort - just see the recent post by Seon Lee who appears to be one of the Mule users: Mule 2.0 + GigaSpaces 6.5 = Pure Sex:

Gigaspaces released 6.5 with API integration with Mule 2.0 … this is just plain awesome. You can use Gigaspaces as the transport (e.g. in place of JMS) and quickly get a SBA up and running utilizing the same concepts I used at RHG when we were servicing B2B problems. You also get the advantage of the clustering ability and fault tolerance that comes with Gigaspaces – which is just pure sex – not to mention all the other great features that come with this advanced Javaspaces implementation (i.e. management tools, monitoring tools, data partitioning, performance features like batching).

I expect to see even more on that line with our latest 6.6 release which includes Seamless Scaling of Web application - check that out!

Scaling Out MySQL

With the recent acquisition of MySQL by Sun, there has been talk about the MySQL open source database now becoming relevant to large enterprises, presumably because it now benefits from Sun's global support, professional services and engineering organizations. In a blog post about the acquisition, SUN CEO Jonathan Schwartz wrote that this is one of his objectives.

While the organizational aspects may have been addressed by the acquisition, MySQL faces some technology limitations which hinder its ability to compete in the enterprise. Like other relational databases, MySQL becomes a scalability bottleneck because it introduces contention among the distributed application components. 

There are basically two approaches to this challenge that I'll touch in this post:

1. Scale your database through database clustering

2. Scale your application, while leaving your existing database untouched by front-ending the database with In-Memory-Data-Grid (IMDG) or caching technologies. The database acts as a persistence store in the background. I refer to this approach as Persistence as a Service (PaaS).

While both options are valid (with pros and cons), in this post I'll focus mostly on the second approach, which introduces some thought-provoking ideas for addressing the challenge.

Disclaimer: While there are various alternative in-memory data grid products, such as Oracle Coherence and IBM ObjectGrid, in this post I'll focus on the GigaSpaces solution, because for obvious reasons I happen to know it better. Having said that, I try to cover the core principles presented here in generic terms as much as possible.

Scaling your database through database clustering:

There are two main approaches for addressing scalability through database clustering:

• Database replication is used to address concurrent access to the same data. Database replication enables us to load-balance the access to the shared data elements among multiple replicated database instances. In this way we can distribute the load across database servers, and maintain performance even if the number of concurrent users increases.

            Limitations:

• Limited to "read mostly" scenarios: when it comes to inserts and updates, replication overhead may be a bigger constraint than working with a single server (especially with synchronous replication)
• Performance: Constrained by disk I/O performance.
• Consistency: asynchronous replication leads to inconsistency as each database instance might hold a different version of the data. The alternative -- synchronous replication -- may cause significant latency.
• Utilization/Capacity: replication assumes that all nodes hold the entire data set. This creates two problems:.1) each table holds a large amount of data, which increases query/index complexity. 2) We need to provision (and pay for) more storage capacity with direct proportion to the number of replicated database instances
• Complexity: most database replication implementations are hard to configure and and are known to cause stability issues.
• Non-Standard: each database product has different replication semantics, configuration and setup. Moving from one implementation to another might become a nightmare.

• Database partitioning ("sharding"): database shards/partitions enable the distribution of data on multiple nodes. In other words, each node holds part of the data. This is a better approach for scaling both read and write operations, as well as more efficient use of capacity, as it reduces the volume of data in each database instance.

          Limitations:

• Limited to applications whose data can be easily partitioned.
•  Performance: we are still constrained by disk I/O performance
•  Requires changes to data model: we need to modify the database schema to fit a partitioned model. Many database implementations require that knowledge of which partition  the data resides in is exposed to the application code, which brings us to the next point.
•  Requires changes to application code: Requires different model for executing aggregated queries (map/reduce and the like).
•  Static: in most database implementations, adding or changing partitions involves down-time and re-partitioning.
•  Complex: setting-up database partitions is a fairly complex task, due to the amount of moving parts and the potential of failure during the process.
•  Non-standard: as with replication, each database product has different replication semantics, configuration and setup. Partitioning introduces more severe limitations, as it often requires changes to our database schema and application code when moving from one database product to another.

Time for a change  -  is database clustering the best we can do?

The fundamental problems with both database replication and database partitioning are the reliance on the performance of the file system/disk and the complexity involved in setting up database clusters. No matter how you turn it around, file systems are fairly ineffective when it comes to concurrency and scaling. This is pure physics:  how fast can disk storage be when every data access must go through serialization/de-serialization to files, as well as mapping from binary format to a usable format? And how concurrent can it be when every file access relies on moving a physical needle between different file sectors? This puts hard limits on latency. In addition, latency is often severely affected by lack of scalability. So putting the two together makes file systems -- and databases, which heavily rely on them -- suffer from limited performance and scalability.

These database patterns evolved under the assumption that memory is scarce and expensive, and that network bandwidth is a bottleneck. Today, memory resources are abundant and available at a relatively low cost. So is bandwidth. These two facts allow us to do things differently than we used to, when file systems were the only economically feasible option.

Scaling through In Memory Caching/Data Grid

It is not surprising that to enhance scalability and performance many Web 2.0 sites use an in-memory caching solution as a front-end to the database. One such popular solution is memcached. Memcached is a simple open source distributed caching solution that uses a protocol level interface to reference data that resides in an external memory server. Memcached enables rudimentary caching and is designed for read-mostly scenarios. It is used mainly as an addition to the LAMP stack.

The simplicity of memcached is both an advantage and a drawback. Memcached is very limited in functionality. For example, it doesn't support transactions, advanced query semantics, and local-cache. In addition, its protocol-based approach requires the application to be explicitly exposed to the cache topology, i.e., it needs to be aware of each server host, and explicitly map operations to a specific node. These limitations prevent us from fully exploiting the memory resources available to us. Instead, we are still heavily relying on the database for most operations.

Enter in-memory Data Grids.

In-memory data grids (IMDG) provide object-based database capabilities in memory, and support core database functionality, such as advanced indexing and querying, transactional semantics and locking. IMDGs also abstract data topology from application code. With this approach, the database is not completely eliminated, but put it in the *right* place. I refer to this model as Persistence as a Service (PaaS). I covered the core principles of this model in this post. Below I'll respond to some of the typical questions I am asked when I present this approach.

How Persistence as a Service works?

With PaaS, we keep the existing databases as-is: same data, same schema and so on. We use a "memory cloud" (i.e., an in-memory data grid) as a front-end to the database. The IMDG loads its initial state from the database and from that point on acts as the "system of record" for our application. In other words, all updates and queries are handled by the IMDG. The IMDG is also responsible for keeping the database in sync. To reduce performance overhead, synchronization with the database is done asynchronously. The rate at which the database is kept in sync is configurable.

The in-memory data model can be different from the one stored in the database. In most cases, the memory-based data model will be partitioned to gain maximum scalability and performance, while the database remains unchanged.

How does PaaS improve performance compared to a relational database?

Performance gains over relational databases are achieved because: 

• PaaS relies on memory as the system of record, and memory is significantly faster and more concurrent than file systems.
• Data can be accessed by reference, i.e., no need for continuous serialization of data, as with a file system.
• Data manipulation is performed directly on the in-memory objects. Complex manipulation is easily achieved by running either Java/.Net/C++ code or a SQL query. There is no need for serialization/de-serialization of data or network calls during the process.
• Reduced contention: instead of placing all data in a single table, and consequently having many clients accessing that table, we split it into many small tables, each of which will be accessed by a smaller number of clients.
• Parallel aggregated queries: queries that need to span multiple partitions to perform join/sum/max operations can be executed in parallel across the nodes. The fact that the queries run on smaller data sets reduces the time it takes to perform the actual operation on each node. In addition, the fact that queries execute on multiple machines leverages the full CPU and memory power of those machines.
• In-process local cache: read-mostly operations are cached in the client application local address space. This means that subsequent reads will be executed locally.
• Avoid Object-Relational Mapping (ORM): read operations are performed directly from memory in object format. Thus, there is no need for O/R mapping overhead at this level. O/R mapping happens in the background either during the initial load process, or during the asynchronous update of the database.

If you keep the database in sync, isn't your solution limited by database performance? 

No. Because:

• Data is sent asynchronously and in batches
• Updates are performed in parallel by all partitions.
• Updates to the database are executed collocated in the same machine as the database through a mirror service. This enables to reduce the network overhead to the data base as well as benefit from specific optimization such as batch operations.
  
• The database is not used for high availability purposes. This means that In-flight transactions are not stored in the database, only the end result of the business transactions. This reduces the amount of updates that are sent to the underlying database. Also keep in mind that queries don't really hit the database, only updates and inserts. All this together means that the IMDG acts as a smart buffer to the database. It is common that the number of read/update hits the IMDG receives is 10x higher than the number of hits on the underlying database is seeing. 
• The database and the application are now decoupled, giving you more options for optimization. For example, there are scenarios where writing to the database is required to ensure the durability of the data.  In this scenario, you can store the data directly in a persistent log (to ensure durability). The log can be updated at a relatively high rate. You can read the data from the persistent log back into the database as an off-line operation. With these options in place we can  easily get to 30,000 to 40,000 updates per second with a single instance of MySQL. If this is not sufficient you can always combine data base clustering  to speed up the data  base  access.

Doesn't asynchronous replication mean that data might be lost in case of failure?
No, because asynchronous replication refers to the transfer of data between the IMDG and the database. The IMDG, however, maintains in-memory backups that are synchronously updated. This means that if one of the nodes in a partitioned cluster failed before the replication to the underlying database took place, its backup will be able to instantly continue from that exact point.

What happens if one of my memory partitions fails?
The backup of that partition takes over and becomes the primary. The data grid cluster-aware proxy re-directs the failed operation to the hot backup implicitly. This enables a smooth transition of the client application during failure -- as if nothing happened. Each primary node may have multiple backups to further reduce the chance of total failure. In addition, the cluster manager detects failure and provisions a new backup instance on one of the available machines. 

What happens if the database fails?
The IMDG maintains a log of all updates and can re-play them as soon as the database becomes available again. It is important to note that during this time the system continues to operate unaffected. The end user will not notice this failure!  

How do I maintain transactional integrity?
The IMDG supports the standard  two-phase commit protocol and XA transactions. Having said that, this model should be avoided as much as possible due to the fact that it introduces dependency among multiple partitions, as well as creates a single point of distributed synchronization in our system. Using a classic distributed transaction model doesn't take advantage of the full linear scalability potential of the partitioned topology. Instead, the recommended approach is to break transactions into small, loosely-coupled services, each of which can be resolved within a single partition. Each partition can maintain transaction integrity using local transactions. This model ensures that in partial failure scenarios the system is kept in a consistent state. 

How is transactional integrity maintained with the database?
As noted above, distributed transactions might introduce a severe performance and scalability bottleneck, especially if done with the database. In addition, attempting to execute transactions with the database violates one of the core principles behind PaaS: asynchronous updates to the database. To avoid this overhead, the IMDG ensures that transactions are resolved purely in-memory and are sent to the database in a single batch. If the update to the database fails, the system will re-try that operation until the update succeeds. 

What types of queries are supported?

• Template matching (matching object based on class name, class hierarchy, and attribute values)
• SQL – support range queries, 'like' semantics, etc.
• Continuous queries – through a combination of notification and SQL.
• Parallel query (a.k.a Map/Reduce) – queries that are not designated for a specific partition are automatically broadcasted to all partitions and the result is implicitly aggregated on the client side.
• Iterator – iterate through a large result-set of data.
  

• You can find some code snippets of the different query APIs here.

This model relies heavily on partitioning. How do I handle queries that need to span multiple partitions?
Aggregated queries are executed in parallel on all partitions. You can combine this model with stored procedure-like queries to perform more advanced manipulations, such as sum and max. See more details below.   

What about stored procedures and prepared statements?
Because the data is stored in memory, we avoid the use of a proprietary language for stored procedures. Instead, we can use either native Java/.Net/C++ or dynamic languages, such as Groovy and JRuby, to manipulate the data in memory. The IMDG provides native support for executing dynamic languages, routes the query to where the data resides, and enables aggregation of the results back to the client. A reducer can be invoked on the client-side to execute second  level aggregation. See a code example that illustrates how this model works here. 

Can I change these prepared statements and stored procedure equivalents without bringing down the data?
Yes. When you change the script, the script is reloaded to the server while the server is up without the need to bring down the data. The same capability exists in case you need to re-fresh collocated services code on the server-side.  

Do I need to change my application code to use an IMDG?
It depends. There are cases In which introducing an IMDG can be completely seamless and there are cases in which you will need to go through a re-write, depending on the programming model: 

	Nature of Integration with IMDG	Comments/limitations
Hibernate 2nd level cache	Seamless	Best fit for read-mostly applications. Limited performance gain as it still heavily relies on the underlying database.
JDBC	Seamless, but limited	SQL commands written against the IMDG are guarantied to run with other JDBC resources. Doesn't support full SQL 92 and therefore existing applications may require code changes.Recommended for monitoring and administration. Not recommended for application development as it introduces unnecessary O/R mapping complexity.
HashMap	Seamless	Extensions such as timeout and transaction support are available as well.
GigaSpaces Spring DAO	Partially seamless	Abstracts the transaction handling from the code. Domain model is based on POJOs, and therefore, doesn't require explicit changes, only annotations (annotation can be provided through an external XML file). If our application already uses a DAO pattern then it would require changing the DAO. This allows  narrowing down the scope of changes required to use an IMDG-specific interface. This option is highly recommended for best performance and scalability.

What topologies are supported?
Replicated (synchronous or asynchronous), partitioned, partitioned-with-backup.
See details here.

Do I need to change my code if I switch from one topology to another?
No. The topology is abstracted from the application code. The only caveat is that your code needs to be implemented with partitioning in mind, i.e., moving from a central server or a replicated topology to partitioning doesn't require changes to the code as long as your data includes an attribute that acts as a  routing index.

How are IMDGs and PaaS different from in-memory databases (IMDB)?

The relational model itself doesn't prevents us from taking full advantage of the fact that the data is stored as objects in memory. For example, when we use in-memory storage in an IMDG, we don't need the O/R mapping layer. In addition, we don't need separate languages to perform data manipulation. We can use the native application code, or dynamic languages, for that purpose.

Moreover, one of the fundamental problems with in-memory databases is that relational SQL semantics is not geared to deal with distributed data models. For example, an application that runs on a central server and was uses things like Join, which often maintains references among tables, or even uses aggregated queries such as Sum and Max, doesn't map well to a distributed data model. This is why many existing IMDB implementations only support very basic topologies and often require significant changes to the data schema and application code. This reduces the motivation for using in-memory relational databases, as it lacks transparency.

The GigaSpaces in-memory data grid implementation, for example, exposes a JDBC interface and provides SQL query support. Applications can therefore benefit from best of both worlds: you can read and write objects directly through the GigaSpaces API, query those same objects using SQL semantics, and view and manipulate the entire data set using regular database viewers.

Can I use an existing Hibernate mapping to map data from the database to the IMDG?

Yes. In addition, with PaaS, the Hibernate mapping overhead is reduced as most of it happens in the background, during initial load or during the asynchronous update to the database.

Further information on Hibernate support is available here.

Can I use PaaS with .Net or C++ applications?

Yes. Starting with GigaSpaces 6.5 both Hibernate (Java) and nHibernate (.Net) are supported. C++ applications deffer to the default Hibernate implementation. In addition, with GigaSpaces' new integration with Microsoft Excel, .Net users can easily access data in the IMDG directly from their Excel spreadsheets without writing code!

Final words:

While this approach is generic and can be applied to any database product, MySQL is the most interesting to discuss as it is widely adopted by those who need cost-effective scalability the most, such as web services, social networks and other Web 2.0 applications. In addition, MySQL faced several challenges in penetrating large enterprises. With the acquisition of Sun, MySQL becomes a viable option for such organizations, but still requires the capabilities mentioned above to compete effectively with rival databases. The combination of IMDG/PaaS with MySQL provides a good solution for addressing some of the bigger challenges in cloud-based deployments. More on that in a future post.

Who needs standards anyway?

There is an interesting debate taking place on InfoQ: What role will the JCP play in Java's future?

"Alex Blewitt described the Java Community Process (JCP) as dead, likening it to a headless chicken which "doesn't realise it yet and it's still running around, but it's already dead". This touched off a debate over the usefulness of the JCP and how much it will play a role in Java's future"

The traditional role of standards was to define a spec that everyone will comply with, and consequently, an open market around the standards will emerge. This, presumably, will enable customers to choose the best implementation and avoid vendor lock-in.

A quick look into current trends reveals that things like PHP and the Spring Framework, neither of which are formal standards, are being adopted more rapidly than many of the existing formal standards. These frameworks are able to quickly respond to market needs, while traditional formal standards lag behind.

Perhaps more importantly, even in projects that use a formal standard, such as JEE, there is typically heavy use of non-standard, proprietary components that come with the products that implement the standard. So even in these cases, the fact that only a part -- sometimes a small part -- of the project complies with a standard significantly reduces the value of the standard. This is especially true in the Java world.

Alex Blewitt focuses on the reasons why JCP isn't successful, but his arguments are true for other software standards bodies as well. IMO there is a more fundamental problem that goes beyond JCP or any other process. At the heart of it is the fact that the primary focus of such standards bodies is on agreeing on a specific API or protocol. This focus is too low level and is often tied to specific semantics that don't leave enough room for innovation and optimization, and therefore, doesn't encourage healthy competition on the implementation of the standard. This makes it very hard to find common ground for agreement and often leads to ugly compromises, tediously protracted processes, etc. These ugly compromises and prolonged processes lead to dissatisfaction from users, and consequently, to lack of adoption  -- and the vicious cycle continues.

Right now we are experiencing this Catch 22 scenario where we want to have an open market and avoid vendor lock-in, but the process to achieve that openness via standards is somewhat closed -- and basically broken. It is this realization that brought me to ask the question: who needs standards, anyway?

The goal of keeping an open market is an important goal that we should continue to strive for. What we need to do is change our thinking and break away from the traditional approach to the role of standards as we know them today.

Here are some ideas to help address the issues:

Leverage open source communities and product adoption as a way for defining de-facto standards

We don't need any regulatory process that will determine which project can be applicable as a standard or not. Anyone who thinks that they have good idea can just spawn their own project and do pretty much whatever they like with it . Adoption, or lack thereof, will be the measure of relevance of the project. In other words, if you're doing the right things and solving real problems then most likely it will be reflected in higher adoption. Adoption is an excellent tool for measuring success.  To make this tool even more effective, you can also apply measurements of adoption, similar to Google PageRank. The rank could be based on number of downloads, number of references in the blogosphere, the quality of those references, etc. One of the implicit benefits of this model is that it brings back the power to the developer. It also  creates a highly competitive environment that encourage innovation and alternative thinking.

Keep It Loose

Leverage new methodologies, such as the declarative abstraction model (dependency injection), annotations and Aspect-Oriented Programming (AOP) to create a loose plug-in model. With this approach, we can plug-in different implementations that don't necessarily comply with the same API -- or even technology. Mule is a very good example of this approach.

With Mule you can plug-in different protocol implementations, as well as different event sources -- such as JMS, Web Services, JavaSpaces and others -- and inject the event into your businesses logic from any of those sources in the same way. The key is that you can achieve this without changing your business logic every time you add a new data-source, and without forcing the use of a specific event-handling API, such as JMS, for example. Spring Remoting does pretty much the same thing. We can write our Service as a POJO and then use the remoting abstraction to invoke that service through a local-proxy, remote proxy, and enable the dynamic creation of proxies, without enforcing the implementation to bind to a specific protocol (IIOP,JRMP) or model.

Commercial vendors can be involved in this process by contributing back to these open source projects, and through that influence their evolution. A good example for this is MySQL, which is supported by big companies such as SAP, and companies like Google or eBay who contributed large portions of code when they found things that were insufficient to meet their needs. They are not making these contributions for philanthropic reasons. Their interest in doing so is quite straightforward: if it's part of an open source project, it's going to be maintained and tested by an entire community of users, and become a de facto standard.

Now don't get me wrong: this model is far from perfect. One of the biggest challenges is that we will end up with too many options to choose from and too many frameworks that don't work together.

However, if we consider the LAMP stack, and how it evolved without a big brother that controls the process, there is reason to be optimistic that this model could work, and is the best one we have.