Nati Shalom's Blog

The next big thing in big data: fast data

The big data movement was pretty much driven by the demand for scale in velocity, volume, and variety of data. Those three vectors led to the emergence of a new generation of distributed data management platforms such as Hadoop for batch processing and NoSQL databases for interactive data access. Both were inspired by their respective predecessors Google (Hadoop, BigTable) and Amazon (Dynamo DB).

As we move to fast data, there’s more emphasis on processing big data at speed. Getting speed without compromising on scale pushes the limits in the way most of the existing big data solutions work and drives new models and technologies for breaking the current speed boundaries. New advancements on the hardware infrastructure with new flash drives offer great potential for breaking the current speed limit which is bounded mostly by the performance of hard drive devices.

Why using existing RDBMS, NoSQL on top of flash drive is not enough

Many of the existing databases — including more modern solutions such NoSQL and NewSQL — were designed to utilize the standard hard drive devices (HDD). These databases were designed with the assumption that disk access is slow and therefore they use many algorithms such as Bloom filters to save access to disk in case the data doesn’t exist. Another common algorithm is to use asynchronous write to commit-log. A good insight into all the optimizations that are often involved in a single write process on NoSQL databases is provided through the Cassandra Architecture. Let’s have a look at what that entails below:

 

The Cassandra write path

 

What happens when disk speed is no longer the bottleneck?

When the speed of disk is no longer the bottleneck, as in the case of flash devices, then a lot of the optimization turns into overhead. In other words, with flash devices it will be faster and simpler to access the flash device directly for every write or read operation.

Flash is not just a fast disk

Most of the existing use cases use flash devices as faster disk. Using flash as a fast disk was a short path to bypass the disk performance overhead without the need to change much of the software and applications. Having said that, this approach inherits many unnecessary disk driver overhead. So to exploit the full speed potential of flash devices, it would be best to access the flash devices directly from the application and treat flash devices as a key/value store rather than a disk drive.

Why can’t we simply optimize the existing databases?

When we reach a point in which we need to change much of the existing assumptions and architecture of the existing databases to take advantage of new technology and devices such as flash, that’s a clear sign that local optimization isn’t going to cut it. This calls for new disruption.

The next big thing in big data

Given the background above, I believe that in the same way that demand for big data led to the birth of the current generation of data-management systems, the drive for fast data will also lead to new kinds of data management systems. Unlike the current set of databases, I believe that the next generation databases will be written natively to flash and will use direct access to flash rather than the regular disk-based access. In addition to that, those databases will include high performance event streaming capabilities as part of their core API to allow processing of the data as it comes in, and thus allowing real-time data processing.

 

Fast data in the cloud

Many of the existing databases weren’t designed to run in cloud as first-class citizens and quite often require fairly complex setup to run well in a cloud environment.

As cloud infrastructure matures, we now have more options to run big data workload on the cloud. The next generation databases need to be designed to run as a service from the get-go.

To avoid the latency that is often associated with such a setup, the next generation databases will need to run as close as possible to the application. Assuming that many of the applications will run in one cloud or another, it means that those databases will need to have built-in support for different cloud environments. In addition, they will need to leverage dynamic code shipping to pass code with the data, and in this way allow complex processing with minimum network hops.

In-memory databases and data grid are the closest candidates to drive next-gen flash databases

RAM and flash-based devices have much more in common than flash and hard drive. In both RAM and flash, access time is fairly low and wasn’t really the bottleneck; in-memory databases use direct access to the RAM device through key value interface to store and index data.

Those factors makes in-memory based databases more likely to fit into the next generation flash databases.

The combination of in-memory databases and data grid on top of flash devices also allows the system to overcome some of the key capacity and cost per GB limitations of an in-memory based solution. The integration of the two will allow an increase in the capacity per single node to the limit of the underlying flash device rather than to the limit of the size of RAM.

The overall architecture of the integrated solution looks as follows (Source: Gartner):

 

 

As can be seen in the diagram above, the IMC (in-memory computing) layer acts as the front end to the flash device and handles the transactional data access and stream processing, while the flash device is used as an extension of the RAM device from the application perspective.

There are basically two modes of integration between the RAM and flash device.

LRU Mode – In this mode, we use the RAM as a caching layer to the flash device. The RAM device holds the “Hot,” i.e. the most recently used, and the flash device holds the entire set.

Pros: Optimized for maximum capacity.

Cons: Limited query to simple key/value access

Fast Index Mode – In this mode, all the indices in RAM and the data in flash holds the entire set.

Pros: Supports complex querying, including range queries and aggregated functions.

Cons: Capacity is limited to the size of indexes that can be held in RAM.

In both cases, the access to flash is done directly using key/value interface and not through disk drive interface. As both RAM and flash drives are fairly fast, it is simpler to write data to the flash drive synchronously and in this way avoid potential complexity due to inconsistency.

It is also quite common to sync the data from the IMC and flash devices into an external data store that runs on a traditional hard drive. This process is done asynchronously and in batches to minimize the performance overhead.

Direct flash access in real numbers

To put some real numbers behind these statements, I wanted to refer to one of the recent benchmarks that was done using an in-memory data grid based on GigaSpaces XAP and direct flash access using a key/value software API that allows direct access to various flash devices. The benchmark was conducted on several devices as well as on private and public cloud-based services on AWS. The benchmark shows a ten times increase in managed data without compromising on performance.

 

 

 

For full details of the benchmark, see the following white paper:

XAP MemoryXtend -Massive Application Storage Capacity for Real-Time Applications

 

 

 

Industry Survey Shows: Organizations Turn to Stream Processing for their Big Data Needs

In a stream processing model, data is processed as it arrives to the Big Data system. With a batch processing model, only once the data is stored does the system run a variety of batch analytics - typically through a map/reduce style of processing.

 Facebook and Twitter are good examples of stream processing models in use today, as I outlined in a previous post entitled Facebook’s vs Twitter’s Approach to Real-Time Analytics.

 According to a recent survey of approximately 250 respondents, there is a trend moving towards stream processing in order to speed up analytics, with a significant increase in the popularity of this model when compared with last year. The number of organizations planning to use stream processing in 2014 has more than doubled (24%) from last year’s amount (10%).

This goes hand in hand with the fact that real-time analytics is also becoming more mainstream, as I pointed out in my 2014 Predictions.

Another interesting data point from the survey is that roughly 43% of the respondents defined real-time as sub-second and 42% defined it as sub-minute. This difference is quite interesting and could lead to different approaches for implementing stream processing.

For example, real-time is defined differently by Facebook and Twitter. In the case of Twitter, they chose Storm as their event processing engine, allowing them to process events at a sub-second resolution. Meanwhile, Facebook defines real-time at a 30 second batch window, choosing a logging-based approach to fit their need.

Based on the survey, it appears that both approaches are valid and can be applied in correlation to the degree of real-timeliness of your analysis.

 

Facebook Log-Centric Stream Processing	Twitter Event-Centric Stream Processing

In-Memory Data Grids become a more popular choice for real-time processing

According to the survey, 64% of the respondents indicated that they plan to combine In-Memory-based solutions for delivering their real-time analytics processing. This is also consistent with last year survey done by Ventana Research.

 I believe that will see an even bigger movement in this direction, as the cost/performance ratio of In-Memory Data Grids and In-Memory Databases would go down significantly with the combination of RAM and Flash devices, which together provide a fairly compelling solution from both performance and cost ratio to that of other disk-based alternatives.

 

Big Data in the Cloud increased to 56%

New developments in cloud infrastructure, such as the support for bare metal in the new OpenStack releases, as well as support, high memory, flash disk, etc, are removing many of the technical barriers for running I/O intensive workloads like Big Data in the cloud.

Indeed, according to the survey, there is a significant shift towards the use of Big Data in the cloud with 55% of organizations either using or planning to run their Big Data in the cloud in 2014.

 

Where do we go from here? How does this effect GigaSpaces’ future roadmap?

There are multiple areas in the GigaSpaces roadmap that aim to address this demand. 

• Real-Time Processing through Storm Integration - In this project, we integrate Storm on top of a Memory backend for both stream processing and data-store.
• Support for Flash Disk - The integration with flash disk is planned for our upcoming XAP release and will include support for SanDisk, Fusion I/O and other flash disk devices.
• Big Data in the Cloud - through Cloudify - We've been continuously extending our Big Data portfolio support and recently added support for Storm and Cognos, adding to our existing support for Hadoop, MongoDB, Cassandra, ElasticSearch, etc.

 

References

• GigaSpaces Annual Big Data & Cloud Survey 2013
• Eight Cloud and Big Data Predictions for 2014
• GigaSpaces and Storm Part 2: XAP Integration
• GigaSpaces And Storm Part 1: Storm Clouds
• Big-Data-Real-Time-Performance - Enjoying both worlds with one Architecture!
• 2012 Big Data Survey Report
• In Memory Computing (Data Grid) for Big Data

Real-Time Big Data with Storm, Cassandra, and XAP

Real-time Big Data is becoming more and more common in Big Data systems.

One of the most common frameworks used for running real-time Big Data system is Twitter Storm backed by a NoSQL database, as shown in the diagram below:

 

In this architecture, Storm is used for processing data as it comes in and the NoSQL data store is used as the output for that processing as well as for reference data storage.

Challenges

This architecture presents a couple of challenges:

• Performance - Storm runs in-memory, and is therefore set to process large volumes of data at in-memory speed. However, a typical storm architecture needs to interface with a data source for its input and a data source for its output. In this context, the overall performance is now determined not by Storm itself but by the data input and output sources. Quite often these interfaces rely on file-based message queues for streaming, and NoSQL for data storage. These interfaces are at least an order of magnitude slower than Storm itself, and therefore become the limiting factor.
• Complexity - Storm itself consists of several moving parts, such as a coordinator  (ZooKeeper), state manager (Nimbus), and processing nodes (Supervisor). The NoSQL data store also comes with its own cluster management. In addition, a typical Big Data system comes with more components such as the application front end, a reporting database, and more. This makes the process of managing the deployment, configuration, fail-over and scaling of such a system quite complex.

Meeting the Performance Challenge

Given that Storm itself runs in memory, it only makes sense that in order for Storm to run at maximum capacity the streaming and data store interfaces should be implemented in-memory as well, as shown below:

As you can see in the above architecture, we added two interfaces that rely on a built-in Storm plug-in - one for the data inputs and the second for data output. In both cases, the underlying implementation is memory-based, and thus removing the impedance that the previous architecture included.

You can read the full details on how this integration works, including a code example, in DeWayne Filppi's post on integration of in-memory computing with Cassandra.

The second optimization that can be added is to the NoSQL data store.

As most analytics applications tend to be read-mostly, we can speed up access to the NoSQL data store using an in-memory local cache. This architecture speeds up read performance between 10 to 1000 times as outlined by Shay Hassidim in his post on real-time Big Data performance.

Meeting the Complexity Challenge

To meet the complexity challenge, we will use a DevOps automation approach using Cloudify in conjunction with Chef or Puppet.

With this approach, we wrap every service with a deployment recipe that abstracts the underlying details on how to manage Storm, Cassandra, and our in-memory data store. Cloudify uses these recipes to automate the deployment of the entire stack. In this way, you only need to interact with Cloudify for the deployment, configuration, and also scaling and fail-over of your stack rather than separately manage each individual component. In addition, we use Cloudify to abstract the underlying infrastructure. This enables us to use the same deployment recipe across different environments. One for testing, another for production, etc. We can also use the same approach to deploy our apps based on the type of workload. We can use a public cloud for running sporadic workload, and thus leverage the elasticity of the cloud and also enable us to create and scale the environment as well as rip it off completely when done. At the same time, we can choose bare-metal machines for I/O intensive workloads, and so on.

You can read more on how you can set Storm using Cloudify on DeWayne's post on Storm and the cloud.

Cloudify comes with a rich set of built-in recipes for other Big Data services as well, making the integration process an out of the box experience. The main ones are listed below:

Final Notes - Optimizing without Code Change

Many real-time Big Data systems are now based on Twitter Storm and a NoSQL data store such as Cassandra.

In this post I tried to outline how we can optimize this architecture by addressing two areas: performance, and management.

The good news is that all this is possible to achieve seamlessly, without any code change. Let me explain:

In the case of Storm we used built-in plug-ins - Spout for streaming and Trident as the data store interface. In this way we can simply plug our memory-based plug-ins under these two integration points. All our existing Storm business logic would work pretty much the same.

We use the same plug-in approach to integrate our in-memory data store with our NoSQL data. This integration makes the data flow between our real time streaming to our NoSQL storage fairly transparently. In addition, it allows us to plug in different NoSQL data stores such as MongoDB, CouchBase, etc., giving us another degree of flexibility.

The same applies to our management layer. Existing Storm and NoSQL deployments are wrapped into plugged in recipes, and don't require any specific code changes.

Not Mutually Exclusive

We don't have to implement all of the optimizations to gain the benefit of this architecture. Each of the optimization points can be plugged in  independently of the others. For  example, if your main pain point is complexity, you can start by only by adding the DevOps automation first. At the same time if your main pain point is performance, you can use the memory-based plug-ins to speed up processing.

The other advantage of the combined architecture that I haven't discussed in this post is that it provides more flexibility in the degrees of consistency and availability in which you set your system, as I outlined in one of my recent posts on the subject In Memory Computing (Data Grid) for Big Data.

References:

• Big Data Real-Time Performance
• GigaSpaces and Storm Part 2: XAP Integration
• GigaSpaces And Storm Part 1: Storm Clouds
• In Memory Computing (Data Grid) for Big Data
• Check out how GigaSpaces XAP and Cassandra integration works

A Groovy Kind of Java

Today's application stack is a much more complex environment than the traditional JEE one. Moving from a Spring/JEE background to the cloud can be a complicated transition.

In talk given at the San Francisco Java Meetup Group i tried to outline how Groovy can help you map and translate an old JEE evironment to a new cloud-based one using  Cloudify in a way that anyone can adapt to their own needs. You can find the slides from the talk here.

GigaSpaces Cloudify & VMware Cloud Foundry the new PaaS Jailbreaker

I was reading Krishnan Subramanian's post, Two Events That “Clouded” Our Thinking In 2011. The thing that caught my attention was Krishnan's comments on why PaaS is a superior alternative to DevOps: 

The shift in the thinking about the enterprise cloud consumption also poured water into the “DevOps” concept advocated by vendors and pundits with their foot in the IaaS world. When organizations embrace PaaS instead of infrastructure services, we don’t need the DevOps marriage and the associated cultural change (believe me, this cultural change is giving sleepless nights to many IT managers and some consultants are even making money helping organizations realize this cultural change). With PaaS, organizations can keep the existing distinction between the Ops and Dev teams without worrying about the cultural change. In fact, with cloud computing, the role of the Ops is not going away but it stays in the background offering an interface which developers can manage themselves.

Krishnan represents one of the common attitudes and subjects of debate between two main paradigms for developing and managing applications on the cloud:

• PaaS -- PaaS takes a developer, application-driven approach. A PaaS platform provides generic application containers to run your code. The PaaS platforms deals with all the operational aspects needed to run your code such as deployment, scaling, fail-over, etc.
• DevOps -- DevOps takes a more operations-driven approach. With DevOps, you get tools to automate your operational environment through scripts and recipes, and keep full visability and control over the underlying infrastructure.

The Difference Between PaaS and DevOps

Both PaaS and DevOps aim toward the same goal -- reducing the complexity of managing and deploying applications on the cloud. But they take a fairly different approach to deliver on that promise.

Christoper Knee summarized it in his blog post DevOps and PaaS -- Friend or Foe? as the difference between Developers and SysAdmin:

• Developers may ask: "if I have a self-service portal for deploying applications (aka PaaS), do I need SysAdmins at all?"
• SysAdmins may ask: "isn't PaaS just a monstrous black box that prevents me from provisioning the specific services we need to deploy real-world apps?" ... The typical SysAdmin thinks that they can get to 75% of PaaS functionality with DevOps tools like Chef without giving up any systems architecture flexibility.

In one of my earlier post on the subject, Productivity vs. Control tradeoffs in PaaS, I tried to outline the main limitation of most of the current "blackbox PaaS" implementations:

I thought that Carlos Ble's post Goodbye Google App Engine (GAE) is a good example that illustrates why the initial perception behind GAE as a simple platform that provides extreme productivity can be completely wrong.

...developing on GAE introduced such design complexity that working around it pushes us 5 months behind schedule.

Part of the reason that brought Carlos through that experience IMO is that in the course of trying to make GAE extremely productive the owner made the platform too opinionated, to the point where you lose all the potential productivity gains by trying to adopt their model. In addition, with a platform like GAE you have very little freedom to leverage existing frameworks such as your own database, or messaging system, or any other third-party service that can in itself be a huge contributor to productivity.

Instead, you're completely dependent on the platform provider's stack and pace of development, and that in itself can work against agility and productivity in yet another dimension. In this specific example, Carlos couldn’t use a specific version of a Python library that would have made his productivity higher, and instead had to work around issues that were already solved elsewhere. This is a good example how the lack of flexibility leads to poorer productivity even in the case of simple applications.

Putting DevOps & PaaS togatehr 

It looks like more people in the the industry have come to recognize that rather than looking at DevOps and PaaS as two competing paradigms, it might be best to combine the two, as Christoper Knee pointed out in his post:

 

What if you could get a PaaS that wasn't a black box, enabling developers to deploy apps easily while still giving SysAdmins the ability to provision any services they needed (a la Cloud Foundry)?

I also came to that realization myself, as I pointed out in my  2012 predictions: 

In 2012, we’ll see many of the DevOps tools, such as Chef and Puppet, integrated into application platforms, making it easier to deploy complex applications onto the cloud. In the same way, we’re going to see more Application Platforms adopting the automation and recipe model from the DevOps world into the application platform. The latter have the potential to transform the opinionated PaaS offerings as we know them today, with Heroku and GAE leading that trend, into a more open PaaS offering that better fits the way users develop apps today, and provide more freedom to choose your own stack, cloud, and application blueprint.

What Makes a Cloudify and CloudFoundary a PaaS Jailbreaker?

The CIO Maginze article Cloud computing disrupts the vendor landscape defines a new class of PaaS platforms that I think represent the definition for a DevOps PaaS:

A PaaS that allows developers to use whatever tool they want to build their cloud applications and the platform tackles the deployment, scaling and management of these apps in the cloud data center.

VMware CloudFoundry is one of the more notable references in that category. Quoting Christoper Knee:

CloudFoundry runs anywhere, incuding on your laptop. CloudFoundry's service container concept is particularly strong, kind of an appliance on steroids.

These ideas were the founding concept behind Cloudify, i.e. putting DevPops & PaaS togather in a single framework. As with CloudFoundry, Cloudify enables you to break away from the "blackbox PaaS". However, even though CloudFoundry is significantly more open than most other PaaS alternatives, at the core it is still based on the "my way or the highway" approach (aka opininated architecture), which forces you to fit into a specific blueprint that is mandated by the platform. Cloudify, on other hand, pushes the envelope even further by adopting the concpet of recipes that was first introduced by DevOps frameworks such as Chef and Puppet. It introduces  more application-driven recipes through the introduction of Domain Specific Language that extends upon the groovy language.

The Cloudify recipes give you the full power to plug in any application stack on any cloud (including a non-virtualized environment) and manage them in similar way to the way you would manage them in your own datacenter or machines. You can also call Chef and Puppet from within the recipes. All this, without hacking the framework itself. As with other similar DSLs, the Cloudify DSL was designed to express even the most complex application management tasks, such as <recovery from a data center failure> in a single line and avoid all the verbose script and API calls that are often involved when you work at the infrastructure level.

All this makes Cloudify an even more open alternative that fits a large variety of the current enterprise application stacks, including:

1. JEE applications
2. Big Data applications
3. Multi-tier applications
4. Native applications (C++,..)
5. .Net, Ruby, Python, PHP applications
6. Multi-site applications
7. Low-latency applications (that can't run on VMs)

It also makes Cloudify more open for special customization of the existing stack, like in the case of:

1. An application that needs a certain version of MySQL (not the one that comes with the framework)
2. An application that needs to run on Redhat (not Ubuntu). or even more interesting -- a case where there are mutiple applications, each needing a different OS served at the same time.

Cloudify also provides a more advanced level of control geared for mission-critical apps, including:

1. Monitoring the application stack and topology
2. Adding custom application metrics
3. Adding custom SLAs

It can also work on wide variety of cloud environments, including Microsoft Azure and non-virtual enviroments.

One of the great powers of the recipe is that it is a great collaboration tool. Once you develop a recipe, it is very easy to share it wtih different groups -- whether internal groups like developement, QA, and operations, where the recipie provides a programmatic definition of thier environment, or it can be shared between the product team and the pro-services team, where your sales and pro-services can easily install and update product versions in a consistent way, as well as reproduce customer scenarios and share them with the support team. Recipes are also a great tool for collaboration through a wider community network, where users can collaborate by sharing common recipes and best practices over the web.

Quick Introduction to Cloudify Recipes

Below is a short snippet that shows a simple application recipe, of a typical java-based web application based on JBoss as the web container and MySQL as a database. The application recipe describes the services that comprise the application, and thier dependencies. The details on how to run MySQL and JBoss is provided in a seperate recipe for each of the individual services. A more detailed description of how a service recipe would look like can be seen here. 

application {
       name="simple app"

       service {
              name = "mysql-service"
      }
       service {

              name = "jboss-service"
              dependsOn = ["mysql-service"]
       }
}

To get a taste, you can try one of the available recepes such as Cassandra, MongoDB, Tomcat, JBoss, Solar, etc., just as a simple way to try out these products on your own desktop or on any of the supported clouds, without the hassle that is often involved in doing so and without even direct relation to Cloudify per-se.

 

References

• Productivity vs. Control tradeoffs in PaaS
• DevOps and PaaS - Friend or Foe?
• PaaS shouldn’t be built in Silos
• Two Events That “Clouded” Our Thinking In 2011
• Cloud computing disrupts the vendor landscape
• Cloudify docs

Big Data Application Platform

It's time to think of the architecture and application platforms surrounding "Big Data" databases. Big Data is often centered around new database technologies mostly from the emerging NoSQL world.  The main challenge that these databases solve is how to handle massive amount of data at a reasonable cost and without poor performanc - distributed databases emerged to address this challenge and today we're seeing high adoption rate and quite impressive success stories such as the Netflix use of Cassandra/DataStax solution.  All that indicate the speed in which this market evolves.

 

 The need for a Big Data Application Platform

Application platforms provide a framework for making the development of applications simpler. They do this by carving out the generic parts of applications such as security, scalability, and reliability (which are attributes of a 'good' application) from the parts of the applications that are specific to our business domain.

Most of the existing application platforms such as Java EE and Ruby on Rails were designed to work with centralized relational databases in mind. Clearly, that model doesn’t fit well to the Big Data world simply because it wasn’t designed to deal with massive amount of data in first place. In addition to that, frameworks such as Hadoop are considered too complex as noted in VP/Research Director for Forrester Research Mike Gilpin, in his post, "Big Data" technology: getting hotter, but still too hard:

"Big Data" also matters to application developers - at least, to those who are building applications in domains where "Big Data" is relevant. These include smart grid, marketing automation, clinical care, fraud detection and avoidance, criminal justice systems, cyber-security, and intelligence.

One "big question" about "Big Data": What’s the right development model? Virtually everyone who comments on this issue points out that today’s models, such as those used with Hadoop, are too complex for most developers. It requires a special class of developers to understand how to break their problem down into the components necessary for treatment by a distributed architecture like Hadoop. For this model to take off, we need simpler models that are more accessible to a wider range of developers - while retaining all the power of these special platforms.

Other existing models for handling Big Data such as Data Warehouse don’t cut it either, as noted in Dan Woods' post on Forbes, Big Data Requires a Big, New Architecture:

...to take maximum advantage of big data, IT is going to have to press the re-start button on its architecture for acquiring and understanding information. IT will need to construct a new way of capturing, organizing and analyzing data, because big data stands no chance of being useful if people attempt to process it using the traditional mechanisms of business intelligence, such as a data warehouses and traditional data-analysis techniques.

To effectively write Big Data applications, we need an Application Platform that would put together the different patterns and tools that are used by pioneers in that space such as Google, Yahoo, and Facebook in one framework and make them simple enough so that any organization could make use of them without the need to go through huge investment.

Here's my personal view on how that platform could look like based on my experience covering the NoSQL space for a while now and through my experience with GigaSpaces.

Characteristics of Big Data Application Platform

As with any Application Platform, a Big Data application platform needs to support all the functionality expected from any application platform such as scalability, availability, security, etc.

Big Data Application platforms are unique in the sense that they need to be able handle massive amounts of data and therefore need to come with built-in support for things like Map/Reduce, Integration with external NoSQL databases, parallel processing, and data distribution services and on top of that, they should make the use of those new patterns simple from a development perspective.

Below is a more concrete list of the specific characteristics and features that define what Big Data Application Platform ought to be. I've tried to point to the specific Java EE equivalent API and how it would need be extended to support Big Data application.

Support Batch and Real Time analytics

Most of the existing application platforms were designed for handling of transactional web applications and have little support for business analytics applications.  Hadoop has become the de facto standard for handling Batch processing; Real Time analytics, however, is done through other means outside of the hadoop framework, mostly through an event processing framework, as I already outlined in details during my previous post Real Time Analytics for Big Data: An Alternative Approach.

A Big Data Application Platform would need to make Big Data application development closer to mainstream development  by providing a built-in stack that includes integration with Big Data databases from the NoSQL world, and Map/Reduce frameworks such as Hadoop and distributed processing, etc.

It also needs to extend the existing Transaction processing and Event Processing semantics that come with JavaEE for handling of Real Time analytics that fits into the Big Data world as outlined in the references below:

Making Big Data Application Closer to Mainstream development practices

• Domain model and Data access API

Writing Big Data application is quite different than writing a typical CRUD application to a centralized relational database in many forms. The main difference is with the design of our data domain model, and the API and Query semantics that well use to access and process that data.

Mapping has proven to be a fairly effective approach to map the impedance mismatch between different data models and API’s.  A good reference on that regard is the use of O/R mapping tools such as Hibernate for bridging similar impedance mismatch between Object and Relational models.

Abstraction frameworks such as Spring has also proven to be quite useful means to plug-in different data sources that doesn’t comply to common API through plug-in approach.

To make the development of Big Data Application closer to mainstream development Big Data application platform need to come with similar mapping and abstraction tools that would make the transition from the existing API’s significantly smoothers.

Today we already have various mapping and abstraction framework available.

For Big Data Mapping tools:

For batch processing were already seeing the increase adoption of frameworks such as Hive which provide an SQL like façade for handling complex batch processing with hadoop.

JPA provides more standard JEE abstraction that fits into Real Time Big Data application.    Google App Engine uses Data Nucleus on top of Big Table, With GigaSpaces we came up with a high performance JPA abstraction for our In Memory Data Grid using Open JPA, and Redhat came up with Hibernate OGM (Object-Grid Mapping). 

In addition to these, we're seeing that the existing mapping tools add extensions to support data partitioning semantics through annotations as in the case of EclipseLink and others.

For Big Data Spring Abstraction:

SpringSource came out with an interesting and even more high level abstraction known as Spring Data that makes it possible to map different data stores of all kinds into one common abstraction through annotation and plug-in approach. The abstraction is leaky currently, but shows progress in the space.

• Business logic

In Java EE, business logic refers to the parts in our application that are responsible for processing the data. As with JavaEE the data often lives in a centralized database SessionBean was often mapped to a single instance per user session.

With BigData the common pattern for processing data is through Map/Reduce. Map/Reduce was designed to handle the processing of massive amount of data through moving the processing logic to the data and distributing the logic in parallel to all nodes.  Having said that, developing parallel processing code is considered fairly complex.

A Big Data Application Platform would need to make Map/Reduce and parallel execution simple. One way of doing that is by mapping this semantics into existing programming models. One example would be to extend the current SessionBean model to support this sort of semantics (as with the GigaSpaces Remoting semantics) – this makes parallel processing look like a standard remote method invocation. Another way is to provide more native parallel execution semantics by extending existing semantics such as createNativeQuery in JPA, or the executor API in Java. 

Interestingly enough, at the time of writing this post I came across  Robin van Breukelen's post, "Distributed Fork Join," that shows how you can use the latest Java 7 fork/join API in similar context.

• Support new semantics that fits the dynamic web era

Clearly, SQL is a great query language (or else it wouldn't have taken over for database access!) but SQL also has its limits. When we're writing web content, we often deal with dynamic data structures that continuously evolve. When the amount of this data gets massive, as with Big Data, it becomes almost impossible to manage those changes through a standard SQL schema evolution model.

To address this need, BigData platforms need to add support schemaless semantics as a first class citizen. That often means that the data mapping layer mentioned earlier would need to be extended to support document semantics. An example for that is MongoDB, CouchBase, Cassandra and the new GigaSpaces Document API (which, as it turns out, maps to MongoDB and Cassandra very easily.)

• Provide built-in semantics for handling of the tradeoffs between consistency, availability, scalability rather than trying to force a least common denominator as with XA and JTA

JEE application platform forced a fairly strict consistency model through JTA, Distributed Transactions, and so forth. Unlike Java EE, Big Data Application platforms need to support more relaxed versions of those semantics that would enable flexibility between consistency, scalability, and performance. A good example on how that configuration may look like is covered in Cassandra/DataStax documentation and one of my previous post on the subject of CAP theory.

• Provide built in support for In-Memory data store to gain best performance and latency

RAM-based devices provide the best performance/latency results. Big Data platforms need to provide seamless integration between RAM and Disk based devices where data that is written in RAM would be synched into Disk asynchronously. In addition to that, they need to provide common abstractions that enable users to use the same Data access API for  both devices and thus make it easier to choose to the right tool for the job without changing the application code.

Good starting points on that regard are the JPA abstractions now being supported by In-Memory Data-Grids and NoSQL data stores as well as the Spring Data abstraction as noted above.

• Provide built-in support for event driven data distribution using pub/sub model

The current Java EE frameworks provide limited support for event processing through message-driven beans and JMS. For Big Data applications, we need to add data awareness to the current MDB model that make it easy to route messages based on data affinity and content of the message. We also need to provide more fine grained semantics for triggering events based on data operation (delete, update,..) as well as content as with CEP (Complex Event Processing). 

In addition to that we need to provide higher level abstractions that use the underlying pub/sub support to enable data synchronization and locality. A good example for that is the use of LocalCache and LocalView where LocalCache is useful for random access patterns by caching the data that was used most recently and LocalView which provides means to Cache a specific segment of data that is known in advance. Both LocalCache and LocalView use the underlying pub/sub support to ensure continuous synchronization of the data from the BigData pool by tracking changes on the Big Data pool and updating the local copy of the data implicitly.

Built in support for public/private cloud

Big Data applications tend to consume lots of compute and storage resources.  There are a growing number of cases where the use of the cloud enables significantly better economics for running  Big Data applications. To take advantage of those economics, Big Data Application Platforms need to come with built-in support for public/private clouds that will include seamless transition between the various cloud platforms through integration with frameworks such as JClouds. Cloud Bursting provides a hybrid model for using cloud resources as spare capacity to handle load. To effectively handle Cloud Bursting with Big Data well have to make the data available for both the public and private side of the cloud under reasonable latency – which often requires other services such as data replication.

Open & Consistent management and orchestration across the stack

A typical Big Data application stack includes multiple layers such as the database itself, the web tier,  the processing tier, caching layer, the data synchronization and distribution layer, reporting tools, etc. One of the biggest challenge is that each of those layers come with different management, provisioning, monitoring and troubleshooting tools. Big Data applications tend to be fairly complex by their very nature; the lack of consistent management,  monitoring and orchestration across the stack makes the maintenance and management of this sort of application significantly harder.

In most of the existing Java EE management layers, the management application assumed control of the entire stack. With Big Data applications, that assumption doesn’t apply. The stack can vary quite significantly between different application layers therefore the management layer of Big Data Application Platform needs to come with a more open management that could host different databases, web-containers etc., and provide consistent management and monitoring through this entire stack.

Final words

Java EE application servers played an important role in making the development of database-centric web application closer to mainstream. Other frameworks such as Spring and Ruby on Rails later emerged to increase the development productivity of those applications. Big Data Application Platforms have a similar purpose – they should provide the framework for making the development, maintenance and management of Big Data Applications simpler. In a way, you could think of Big Data Application platforms as a natural evolution of the current application platforms.

With the current shift of Java EE application platforms toward PaaS, we're going to see even stronger demand for running Big Data applications on cloud based environments due to the inherent economic and operational benefits. Compared to the current PaaS model, moving data to the cloud is more complex and would require more advance support for data replication across sites, cloud bursting etc.

The good news is that Big Data Application platforms are being implemented with these goals in mind, and you can already see migration yielding the benefits one should expect.

References

• NoSQL Job Trends – August 2011 (Dzone)
• Big Data" technology: getting hotter, but still too hard (Forrester)
• Big Data Requires a Big, New Architecture (Forbes)
• Real Time analytics for Big Data: Facebook's New Realtime Analytics Syste
• Example for Big Data application using Cloudify

Speaking of Score Card..

They say that timing is everything in life.. this time was no exception.

Yesterday I published my score card for 2nd generation PaaS and how it differs from Application Servers.

A few hours later, I received an email with a presentation from one of our new customers that is going to present at our GigaSpaces Roadshow.

This slide caught my attention:

 

* Note that GS stands for GigaSpaces.

The customer is a large ISV (1900 employee, $3B in revenue...). Their new product is a real-time BI system that needs to process events in real time from multiple data sources -– social media included (most other alternatives do that in batch mode) and produce actionable reports on customer activities. They had fairly challenging scaling, latency, and cost requirements:

• Massive Scale, 100M Customers, a billion events a day
• Business real time < 100ms
• Low cost 0.05 customer/year

Now it is not often that a company the size of GigaSpaces competes with giants like IBM and Oracle on one of their core IPs and leaves them so far behind.

How was this possible?

Well, our advantage is in our disadvantage –- size. Let me explain.

Removing the Silos

GigaSpaces as a case study

As a small company, we have to be extremely efficient at what we do.  We adopted Lean methodology and Scrum across the board, and we stick to that model quite religiously.  Every feature goes through millions of automated tests. We manage to build GA-quality releases every two weeks.

On the product side, we have designed our product in a way that follows the principles that I laid out in my previous post, i.e. -- without Silos. In GigaSpaces everything is mapped to data. Tables, Queues, even method invocations are mapped to the same underlying data structure representation. 

We maintain ONE cluster throughout the entire stack to manage this scalability and high availability of the data.  ALL of our APIs are simple facades on top of our shared data cluster (a.k.a space). This means that when we optimize performance, fix bugs, improve scalability, and add new features -- it applies immediately through the entire stack, including the .Net and C++ parts of our product.

The fact that all of our APIs share the same data opened new opportunities for optimization that weren’t possible before. For example, we can avoid a lot of the synchronization overhead that is associated with passing a message and then updating it. We can also mesh the driftnet APIs  without unnecessary copies.

As an example, I can choose to use the key/value store API for low latency transaction processing and the SQL API for analytics over the same data that was produced through the key/value APIs.

Because of our size, we couldn’t afford the high maintenance and support that often comes with the type of deployments that we deal with. As a result, we were forced to develop lots of built-in automation and troubleshooting tools as an integrated part of our product, and later we integrated it with the development framework and application in such a way that every application is made operationally ready from its inception.

Now, the nice thing in all this is that both we and our customers benefit from that efficiency. We gain more customers without increasing our costs. Our customers get a better quality product, short fix cycles, and more features faster.

The latest 8.0 release is the best evidence of this, IMO. We managed to deliver 3 times more features than what we normally accomplished in previous releases, and all that with only a minor increase in our development team.

 *A Little Promotion*

If you're in London on the 2nd of February , Paris on 3rd, San Francisco on 7th, or NYC on the 9th, I’d like to welcome you to join our Roadshow and hear more first-hand stories from these customers (Dow Jones, Sears, Amdocs, and many more top-tier organizations). See you there!

You can register and read the full details here.

PaaS shouldn’t be built in Silos–Part II–The PaaS Score Card

I met Geva Perry on Friday last week and as in many of our discussions we started exchanging thoughts on various topics that bothers each of us. Not surprisingly we ended up discussing my last post, PaaS shouldn’t be built in Silos, in which Geva had some part of (without even knowing).

Anyway, Geva brought an interesting question – "What would be your scorecard for second-generation PaaS and how would it be different than an App Server scorecard?"

It took me a second to think and I came up with the following answer which I’d like to share with you and get your thoughts on.

The Second-generation PaaS score card

1. Elasticity –  What happens when I add another machine to my application pool? Which parts of my application can grow? Can I control any of this?
2. Multi-tenancy – what are the means provided by the PaaS to handle multi-tenancy of:
   • Data – Can I have multiple customers accessing the same data service but in isolation from one another?
   • Application – Is it possible to share multiple applications on the same hardware? Is it possible to control the various sharing/isolation level between the different Applications? Can an application affect the service levels offered by other applications on the same hardware?
3. Continuous deployment – What is the process for rolling out new applications, features, or services? Will it involve downtime? What is the process for  changing my data schema? What is the level of granularity for deployment - must I deploy new virtual machines, or entire applications as a single unit of deployment, or can I roll out specific application components?
4. Scalability – What is nature of the built-in scalability? How is it implemented? How can I trust the scalability features? How is scalability consistent across the various tiers of your PaaS? What are the tradeoffs in terms of consistency, or API semantics?.
5. DevOps – What are the tools that enable control of the deployment of your app? How robust are those tools – will they handle also the case of post deployment? What are the troubleshooting and monitoring tools that come with the platform? Can I use the same tools in a consistent way in my development and production environment? Can they integrate with other data-center monitoring tools?
6. Middleware services – What are the middleware services that are provided with the platform (Web, Data, Messaging, Analytics)? Is it possible to plug in other middleware services of my own to the platform?  How easy it is to use those services? What are the performance or latency characteristics of those services? How well they are integrated - i.e., does each one come with its own high availability, scalability, and management tools or do they share the same underlying clustering?

The main thing to note is that I scored Middleware Services as the last item. This is intentional.

You could  argue that this is incorrect, as you can’t do anything with the platform without those services - many that I spoke with about this actually scored the middleware services as the first (and most important) item and spent the entire bulk of their design and selection process on that item. They didn’t realize that by focusing on middleware services they often miss out on everything else - the other five items. 

That said, is there a "right" or "wrong" viewpoint here? Well... no. As usual, there is no absolute "right" or "wrong," but there are different perspectives. Let me try and explain the rationale behind my perspective:

My score is based on the assumption that most of the platforms provide similar degrees of features at the individual middleware services level. (Since they usually compete with each other based on a list of features, this is very common.) The differences between platforms tend to be relatively marginal on a feature by feature basis. The real big differences are:

1. How well those services are integrated in compliance with items 1-5 above
2. How easy it is to plug-in my own services
3. How open those services are for different languages and APIs.

Note that item 2 is really an interesting one.

Today, middleware services evolve at an extremely fast pace. Unlike the application server world where you expect to get everything with the platform, with PaaS it is more likely that none of the platforms can deliver every service as they evolve.  (There's a perceptible lag between conception and delivery.)

It is therefore important to see that the platform is geared to host external ecosystem services in the same way it hosts its own pre-packaged services. It's important to show that I can plug in my own services to manage and extend and use the deployment, management, and multi-tenancy aspects that I listed early in this post.

Item 3 is also important.

Java EE application servers came to the world under the assumption that the world is entirely built on Java. While some shops could get away with this assumption - at least internally - it's never been very true. Now, someone presupposing an all-Java architecture is wearing blinders. In the real world, one has to expect multiple languages - externally, at the very least, and more likely internally as well.

One common way in which some of the platforms address the multi-environment need is by  exposing a RESTful interface to their services. For example, Microsoft Azure provides a way to run Java on top of their platform by exposing all their services through REST. That’s fine to some degree but it also means that all other languages becomes second class citizens to your platform if this is the only option, as REST often comes with a performance and latency cost.  That’s probably the reason why some of the platforms support Facebook Thrift in addition to their support for a REST API. In GigaSpaces, we even went further and enabled the binding of .Net, Java and C++ without any network call from the context of a single process.

Test yourself – are you building a PaaS or yet another Application Server?

If you end up with a score card that focuses on middleware services (item six in my scorecard) and your entire application has solutions that fit into the silos of the other items in the stack - possibly comparing compliance to spec XYZ, or asserting that it supports all of the WS-* specification, or what-have-you ... that’s fine, but you should note that you're in the business of building yet another application server, not a PaaS.

We're still in the early PaaS days. I would be very interested to learn about other perspectives, specifically - how do you  plan to select your PaaS platform? How would your score card would look like? What features would you demand or offer as a competitive advantage?

It's important for us, as an industry, to know what issues we are trying to solve, if we're going to solve them correctly as a whole.

 

My JavaOne 2010 Plans

JavaOne 2010 is coming next week. This year will be the first time Oracle is running the show, so it'll be interesting to see how well (or badly) Oracle treats the community, as there has been lots of speculation recently on how Oracle plans to monetize the technology. The recent lawsuit against Google and the public departure of James Gosling the father of Java didn’t helped to dismiss that fear.

My plan for this year is sort of a continuation of last year. Last year, I was mostly focused on the emergence of Cloud/PaaS technologies and how they will shape the next-generation application server stack (see relevant posts here and here). This year I'm going to drill down more into the components that make up this new PaaS/Cloud stack. My session at JavaOne (NoSQL Alternatives: Principles and Patterns for Building Scalable Applications) is going to deal specifically with data scalability patterns as they have emerged through the various NoSQL implementations. In this talk, I'll try to focus on the common principles and patterns behind the various approaches, such as Sharding, Eventual Consistency, Query models, Map/Reduce, and Transaction Processing. I’ll also discuss how the two models SQL/NoSQL can live together using the examples from Google JPA/BigTable, HBase/Hive and GigaSpaces/SQL and the document model.

For those who don’t plan to attend JavaOne, or for those who are just interested in open discussion around PaaS/Cloud and Dev/Ops, we'll be holding two meetups in collaboration with Adrian Cole from JClouds on Tuesday and Wednesday. Both meetups will take place at 6:30 PM at the Joyent headquarters (345 California Street Suite 2000, 15 min walk from Moscone Center) who were kind enough to host the event.

• In the PaaS showdown: Java vs Ruby meetup Wesley Beary and Ed Muller from EngineYard are going to reveal some of the architecture behind their PaaS as well as their experience in moving to Terremark. I’ll expand upon the patterns and ways to build your own multi-tenant PaaS on any cloud. This meeting will also be an interesting opportunity to get both the Ruby and Java camp together and learn and compare between the Java and Ruby stack.
• In the DevOps meetup were going to have few lightning talks on production stories, strategies, and tradeoffs, including decisions such as native Java vs. the rest APIs, pull vs. push provisioning, configuration, and integration, agentless(ssh) vs. agent OS control. Following the lightning talks we'll have an open discussion and Q&A panel based on your feedback.

** Note that the room for these meetups is limited so if you plan to attend please RSVP ASAP so we can plan accordingly.

I marked other sessions and BOFs (33 in total), mostly around my favorite scalability, NoSQL, and Hadoop topics during the JavaOne conference. If you have other suggestions or comments, you're welcome to post them here, and I’ll update the list accordingly.

See you next week…

ID	Title	Type	Presenter(s)	Company
S313726	Performance and Debugging Advancements in OpenJDK	BOF	Eric Caspole; Tom Deneau; Vladimir Kozlov	Advanced Micro Devices; Advanced Micro Devices; Oracle
S314364	Standard Dependency Injection (DI) with @Inject and JSR 330	BOF	Costin Leau	SpringSource
S314377	Performance Tuning with Cheap Drink and Poor Tools	BOF	Kirk Pepperdine	Kodewerk Ltd.
S313904	Comparing Apache CXF and Axis2 Web Service Frameworks	BOF	Thomas Liou; Jim Smith	PDX; PDX
S314196	A New Era for In-Memory Data Grids	BOF	Manik Surtani	Red Hat, Inc.
S314190	The Better RESTful Web Services Framework: Jersey or Spring Framework	BOF	Hien Luu	Netflix
S314196	A New Era for In-Memory Data Grids	BOF	Manik Surtani	Red Hat, Inc.
S314563	Hadoop vs. Relational Database: Shout-out Between a Java Guy and a Database Guy	BOF	Karthik Shyamsunder; Chethan Thippeswamy	VeriSign Inc.; VeriSign
S313430	Chillin' with ICEfaces	BOF	Ted Goddard	ICEsoft Technologies
S313988	The Groovy and Grails BOF, with Live Grails Podcast Recording	BOF	Glen Smith	Bytecode Pty Ltd
S314360	Dependency Injection (DI) Flavors in Spring 3	BOF	Costin Leau	SpringSource
S314648	Give Me a Proper Visual JavaServer Faces Editor or Else . . .	BOF	David Konecny	Oracle
S313773	Going Beyond Memory Leaks: Debugging Java from Dumps, Using Memory Analyzer	Conference Session	Chris Bailey; Trent Gray-Donald	IBM; IBM
S313977	Functional Programming in Java: Lessons Learned by GridGain	Conference Session	Nikita Ivanov	GridGain, Inc.
S314432	Multiple Languages, One Virtual Machine	Conference Session	Brian Goetz; John Rose	Oracle; Oracle
S313223	Practical Big Data Processing with MapReduce and Hadoop	Conference Session	Shevek M	Karmasphere, Inc.
S314192	Measuring Performance and Looking for Bottlenecks in Java-Based Data Grids	Conference Session	Manik Surtani	Red Hat, Inc.
S314235	The Cassandra Distributed Database	Conference Session	Jonathan Ellis	Rackspace Hosting
S314496	Top 10 Lessons Learned from Deploying Hadoop in a Private Cloud	Conference Session	Rod Cope	OpenLogic, Inc.
S313637	What's New in Grails 1.3~	Conference Session	Graeme Rocher	SpringSource
S313876	JSR 299 (CDI), Weld, and the Future of Seam	Conference Session	Dan Allen	JBoss by Red Hat
S314404	A Lean, RESTful Java Architecture for Building Rich HTML5 Web Applications	Conference Session	Keith Donald; Jeremy Grelle	VMware; VMware
S314445	Scale Java Persistence API Applications with OpenJPA Slice	Conference Session	Pinaki Poddar	IBM
S314492	Java Persistence API (JPA) 2.0 with EclipseLink	Conference Session	Doug Clarke; Shaun Smith; Gordon Yorke	Oracle; Oracle; Oracle
S312989	Comparing Groovy and JRuby	Conference Session	Neal Ford	ThoughtWorks Inc.
S312998	NoSQL Alternatives: Principles and Patterns for Building Scalable Applications	Conference Session	Nati Shalom	GigaSpaces, Inc
S314142	Groovy and Concurrency	Conference Session	Paul King	ASERT
S314154	Writing Domain-Specific Languages (DSLs), Using Groovy	Conference Session	Paul King	ASERT
S314226	SomeSQL: Combining NoSQL Technologies with Existing RDBMS Architectures	Conference Session	Joshua Tuberville	eHarmony, Inc.
S314782	Choosing the Right NoSQL Database	Conference Session	Tobias Ivarsson	Neo Technology
S314413	Extracting Real Value from Your Data with Apache Hadoop	Hands-on Lab	Michal Bachorik; Aaron Kimball; Daniel Templeton	Oracle; Cloudera; Oracle
S313963	Script Bowl 2010: A Scripting Languages Shoot-out	Panel	Rich Hickey; Guillaume Laforge; Raghavan Srinivas; Dick Wall; Roberto Chinnici	Clojure; SpringSource; Self; Navigenics, Inc.; Oracle

YeSQL (part II) - Putting NoSQL, SQL and Document Model Together

In my previous post (YeSQL: An Overview of the Various Query Semantics in the Post Only-SQL World ) I introduced the common query semantics in the post Only-SQL world and made an argument that the right approach is to decouple the query semantics from the underlying NoSQL implementation. This would allow us to combine SQL semantics with NoSQL backend to achieve the best of both worlds -- standard query and scalability.

In this post I wanted to illustrate this idea through some code example using GigaSpaces as the underlying implementation of the concept. The example is based on the the same code examples from another earlier post, WTF is Elastic Data Grid? (By Example) where I covered some of the simple models for writing and reading objects to a distributed data grid. Toward the end I will reference other examples such as Datanucleus and Hbase, as well as cover patterns for supporting NoSQL semantics such as document model with existing RDBMS such as MySQL.

SQL/NoSQL Code Example (Using GigaSpaces)

The Data Model

@SpaceClass
public class Data implements Serializable {

private Long id;
private String data;
private Map info;

public Data(long id, String data , Map info ) {

this.id = id;
this.data = data;
this.info = info;

}

@SpaceRouting
@SpaceId
public Long getId() {
return id;
}

// getter/setter for id, data, info attributes omitted

}


As with the previous example, we use @SpaceRouting annotation to set the routing index.  This index determines which target partition this instance should belong to. @SpaceId determines the unique identifier (Key) for this instance.

Adding Document Model Semantics 

Most of the common examples for document-based APIs use a JSON data model as the document. A document in JSON world would look something like this:

Source: Wikipedia

From a NoSQL implementation perspective, a document often translates to a Map of Maps where each attribute is mapped to a key,value representation and nested value to a key whose value is a Map and so forth.

Document Model in GigaSpaces

In the current version of GigaSpaces a document is basically a Map attribute whose values are indexed. That gives the flexibility to add,remove attribute on an existing object without changing the object schema as with any schemaless API. A nested object would be mapped to a key whose value is a complex object that is stored just like any regular POJO.

In this example I added an info attribute that is basically a Map of Key/Values.

private Map info;


One of the main benefit of this approach is the combination of a strongly typed POJO/Table model with the flexibility of the document model. In other words, I could match objects by type, and each object can have a “fixed” structure that must conform to certain type, and a variable part that can vary on a per instance basis.  In our specific example, when I look for a Data object I'm guaranteed to get the Data.id and Data.data, but Data.info could include a variable list of attributes that can vary on a per instance basis. I can still “pin” some attributes by forcing an indexing on the relevant keys as illustrated below:

// this defines several indexes on the same info property
@SpaceIndexes( { @SpaceIndex(path = "info.address", type = SpaceIndexType.BASIC),
@SpaceIndex(path = "info.socialSecurity", type = SpaceIndexType.BASIC)
})
public Map getInfo() {
return info;
}


Note: As of GigaSpaces 8.0 release the document model semantics would be extended to support full hierarchy of Maps of Maps and dynamic indexes.

Combining SQL Query with (GigaSpaces) NoSQL Data Store

Now that we’ve gone through the API example let’s see how we could insert this data into a NoSQL data store (GigaSpaces in this specific case) and query it through SQL.

Inserting the Data Object

for (long i=0;i<1000;i++)
{
gigaSpace.write(new Data(i,"message" + i, createInfo(i)));
}


The createInfo() generates a new Map and fill it with values as can be seen below:

public static Map createInfo(long i)
{
Map< String, Object> info = new HashMap();
info.put("address", i + " Broadway");
info.put("socialSecurity", 1232287642L + i);
info.put("salary", 10000 + i);

return info;
}


Querying the NoSQL document data using SQL

There are basically two models for querying data using SQL in GigaSpaces. (1) Adding SQL-like queries using the GigaSpaces SQLQuery API (2) using a fully standard SQL jdbc driver.

Querying the data using a SQL-like model

Data[] d = gigaSpace.readMultiple(
new SQLQuery(Data.class, "info.salary < 11000 and info.salary >= 10000"),
Integer.MAX_VALUE);


The gigaspaces.readMultiple(..) operation is equivalent to a “select” statement. It takes the class (the equivalent of the from table name clause in SQL), and query clause “info.salary < 11000 and info.salary >= 10000” (the equivalent of the where clause in SQL) .

As we can see the syntax of the SQL query borrows the syntax of an object-oriented model where I can reference to the associated attributes within the document attribute just like any nested attribute. In our example info.salary would point to the info Map and pull the attribute who’s key=”salary”.

This API is useful if you're already working in POJO as your domain model, as it works with objects natively and therefore can bypass the need for any O/R Mapping.

Querying the data using standard JDBC

In this example we will illustrate how we can plug different query engines to the same data instance. In this case we will use a standard JDBC connection to connect to the data in the following way:

Connection conn;
Class.forName("com.j_spaces.jdbc.driver.GDriver").newInstance();
String url = "jdbc:gigaspaces:url:jini://*/*/myElasticDataGrid";
conn = DriverManager.getConnection(url);
Statement st = conn.createStatement();
String query = "SELECT * FROM com.gigaspaces.examples.Data";
ResultSet rs = st.executeQuery(query);
// Iterate through the result set
int i = 0;
while (rs.next()) {
System.out.println("Data [" + (i++) +"] "+ rs.getString("data") );
}


The first line sets up the GigaSpaces JDBC driver. The GigaSpaces JDBC driver is responsible for mapping the SQL query language into a the underlying GigaSpaces methods. That means that from a GigaSpaces data store perspective JDBC calls looks pretty much the same as any other call. The URL uses the following pattern <gigaspaces jdbc prefix>:<gigaspaces space url>, where the gigaspaces-jdbc-prefix is always set to jdbc:gigaspaces. The space URL points to the relevant data grid cluster. 

Note that one of the interesting concepts that comes with this is that you don’t need to point to a specific host as you would in most of the today's databases, but rather we use “*” as the host name, which initiates a network discovery using multicast to find the relevant instances of the cluster.

The rest of the code looks just like any other SQL call. We use fairly basic mapping where every class is mapped to a table and and an object attribute is mapped to a column. Note that unlike complex O/R mapping we leverage the fact that a space can store objects in their native format. That means that we don’t need to break nested objects into different tables but instead we store them as single embedded java object where the relationships are kept consistent with their original java representation.

Since the standard SQL doesn’t support nested object queries our current version of the JDBC provides access to the top level attributes.  Any nested object is treated as a POJO and is matched according to the  POJO based template matching semantics. Currently, this limits the type of queries that one can perform on nested object compared with the SQL-like API because we can’t perform matching on individual fields within the document or any nested object. As the SQLQuery and JDBC layer use the same underlying query engine, this limitation is only a semantic limitation and not a technical limitation, and will be resolved in the next release of GigaSpaces.

What About Performance?

Alex Popescu, co-founder and CTO of InfoQ.com, made a comment in his post NoSQL Databases Should Support SQL Queries questioning the performance overhead associated with adding another layer of indirection, as I pointed out in my original post. He rightfully quoted Jeff Kesselman:

The two software problems that can never be solved by adding another layer of indirection are that of providing adequate performance or minimal resource usage.

Indeed, this is one of the main challenge in this entire discussion. It is relatively simple to add another level of indirection but it's almost impossible to make it perform well.

The key to addressing the performance challenge relies on the implementation of the underlying data store and how well it is suited to support the functionality required by the higher level abstraction. In our specific NoSQL discussion the performance of the SQL abstraction would be greatly influenced by the ability of the underlying NoSQL data store to support complex queries at the core level. In this case, adding a different set of query semantics would be a matter of simple syntax mapping which should yield negligible overhead and in some cases could turn out to be more efficient, as we could support algorithms that are not simple to implement at a lower level. A good example is found in Kevin Weil's presentation Hadoop, Pig, and Twitter (slides 17-18). Kevin provides an example of how a simple Pig query could map to a fairly complex hadoop task in Java. So even if a lower level API could be more efficient at a micro level, it might turn out to be less efficient at the macro level as a result of the associated complexity. 

In our example the abstraction provided by putting Datanucleus on top of Google BigTable is probably going to yield fairly high overhead when it comes to complex queries, because most of the query semantics are implemented at the mapping layer. With GigaSpaces, we chose to support all the query semantics at the core layer and make the SQL abstraction a thin semantics mapping layer. A recent benchmark test showed only 2% difference between JDBC queries and native queries.

 

GigaSpaces Native Query engine makes the overhead 
 of the SQL abstraction negligible

Other SQL/NoSQL References

Datanucleus provides a JPA/JDO mapping layer ontop of various datastore implementations.

As such, it well positioned to provide a common SQL abstraction layer. It is also well advanced in the area of becoming a common mapping layer for multiple NoSQL datastores. Currently it supports HBase, BigTable, Amazon S3. MongoDB and Cassandra are currently works in progress.

As there is still a big difference between the various datasource implementations (each supports only a subset of query semantics), each datasource  plugin  covers a subset of Datanucleus features. There are various documented limitations per datastore. You can find the list of supported features in the relevant datastore plugin documentation (a reference to googlestorage is provided here ).

You can also find a useful code example for using the Datanuclues JPA plugin for Hadoop/Hbase in the following post Apache Hadoop HBase plays nice with JPA by  Matthias Wessendorf.

RDBMS Support for NoSQL Semantics

As I was working on this write-up, which centers around supporting SQL on top of a NoSQL datastore, I came across an interesting post: Schema-Free MySQL vs NoSQL by Ilya Grigorik, CTO/Founder at PostRank.

In his post, Ilya provides an interesting pattern that shows how you can store a schemaless document model on a MySQL datastore.

Instead of defining columns on a table, each attribute has its own table (new tables are created on the fly), which means that we can add and remove attributes at will. In turn, performing a select simply means joining all of the tables on that individual key

Ilya ends with a comment that I thought was in line with the my thoughts outlined in this post:

..there is absolutely no reason why we can’t have many of the benefits of “NoSQL” within MySQL itself.

Final Words

In this post I've tried to illustrate, mainly through the GigaSpaces example, how the SQL and NoSQL models can be brought together to achieve the best of both worlds, i.e., the scalability of the NoSQL model and the rich and standard query semantics of SQL. At the same time, I believe that additional semantics that are not currently supported by SQL such as the document model and object relationship can also be brought together as an extension to current SQL semantics as illustrated in this post.

Having said that, there will be a time where a Object or document-centric language are more intuitive or simpler to use then SQL. The decoupling of the datastore and the query language should enable us to write objects in an Object/Document-centric model and still query it using a SQL engine and vice versa.  This will give us more flexibility to choose the language that best fits the context in which it is being used, and mix and match among them just as we do today in any web application where we mix together different languages such as  HTML, CSS, JavaScript and other languages.

Finally, I believe that large part of the discussion in the NoSQL community took a wrong turn by putting too much emphasize on the query language rather then the scalability patterns which IMO should remain the only motivation for switching between one datastore to the other.

References

• YeSQL: An Overview of the Various Query Semantics in the Post Only-SQL World
• WTF is Elastic Data Grid? (By Example)
• Apache Hadoop HBase plays nice with JPA by  Matthias Wessendorf
• Hadoop, Pig, and  Twitter presentation by Kevin Weil 
• Schema-Free MySQL vs NoSQL by Ilya Grigorik
• NoSQL databases Should Support SQL Queries by Alex Popescu