Nati Shalom's Blog

Marrying memcached and NoSQL

Abstract:

Memcached is one of the most common In-Memory cache implementation.  It was originally developed by Danga Interactive for LiveJournal, but is now used by many other sites as a side cache to speed up read mostly operations. It gained popularity in the non-Java world, too, especially since it’s a language-neutral side cache for which few alternatives existed.  

As a side-cache, Memcache clients relies on the database as the system of record, The database is still used for write,update and complex query operations.  Since the  memcached specification includes no query operations, memcached is not a database alternative, unlike most of the NoSQL offerings. It also exclude memcache from being a real solution for write scalability. As a result of that many of the heavy sites started to move away from Memcache and replace it with other NoSQL alternatives as noted in a recent highscalability post MySQL And Memcached: End Of An Era?

The transition away from memcached to NoSQL could represent a large investment as many sites are already heavily invested in memcached usage. In this post, I'll illustrate an alternative approach in which we’ll extend the use of memcache for write scaling, add other goodies such as high availability and elasticity by plugging GigaSpaces as the backend datastore, and avoid the need for a re-write. The pure Java implementation could also be seen as a benefit as it can increase the adoption of memcached within the Java community and leverage the portability of java to other platforms,

Memcached overview

The diagram below shows a typical use of Memcache. It outlines a simple deployment topology often referred to as a side cache. This topology is very popular to address read mostly scenarios.

Typical use of Memcache for scaling read mostly applications

In this mode, read operations first check the memcached servers for an artificial key, derived from the requested data. If the key doesn’t exist on the memcached server, a query is made to the database, and the result is then stored with that artificial key into the memcached instances. Subsequent calls with that query go through the memcached deamon and thus saves access to the database. Updates remove the data from the memcached instances, then write the data to the database and the memcached instance at the same time in the most common situation. Wikipedia has a code snippet that provides a more detailed example of this in action.

The main benefit of this model lies in its simplicity. The memcached API is extremely simple to use for this specific scenario. The fact that it relies on a simple and open protocol as opposed to a rich client bound to to a specific language and implementation makes it portable across a wide variety of languages and environments. It also known to be fairly scalable for handling read operations as due to the inherited share nothing model.

The advantages of memcached that make it so simple and popular are the exact same things that make it fairly limited for any scenario that goes beyond read-mostly/side-cache scenarios. Some of those limitations were the main driver that forced some popular sites to switch to a NoSQL as an alternative to memcached as noted by Digg and Twitter.

From Looking to the future with Cassandra:

The fundamental problem is endemic to the relational database mindset, which places the burden of computation on reads rather than writes. This is completely wrong for large-scale web applications, where response time is critical.

From Cassandra @ Twitter: An Interview with Ryan King:

We have a lot of data, the growth factor in that data is huge and the rate of growth is accelerating. We have a system in place based on shared mysql + memcache but its quickly becoming prohibitively costly (in terms of manpower) to operate. We need a system that can grow in a more automated fashion and be highly available.

Summarizing the points from Digg and Twitter, the main limitations of the current Memcached implementations are the lack of support for:

• Write scalability
• Elasticity
• High availability

I would also add to that list the lack of a consistency model, the limited query support, a client-based optional sharding model, management, and monitoring.

Marrying memcached and NoSQL

Many of the limitations of memcached outlined above are addressed by various NoSQL/In-Memory-Data-Grid implementations. The main difference is that many of the NoSQL alternatives were designed to act as a database and not just as a side-cache. As such they were geared for write-scaling and come with built-in elasticity and high availability in mind.

The main motivation behind the integration between the two is to address the current limitations of memcached without forcing a complete and expensive re-write. Another motivation is that we can leverage the rich client and language support that comes along with the memcached protocol as a simple way to extend the use of NoSQL alternatives to other languages.  

While the pattern of integrating Memcache with existing NoSQL backend is fairly generic, in this post i would refer specifically to the GigaSpaces integration.

GigaSpaces memcached support

Joseph Ottinger published a good summary of the rationale behind the GigaSpaces memcache support in a post on TheServerSide.com, “Did Someone Say GigaSpaces Now Has Memcached Support?” In this post, I'll try to provide a reference to the underlying GigaSpaces implementation in the context of the memcached usage pattern that I outlined above.

The GigaSpaces memcached support is fairly simple. It consists of a listener service (written in Java) that implements the memcached protocol for client applications, and maps the protocol into equivalent GigaSpaces operations. There are three mode of operations in which the integration works:

Memcached-compatible – reliability and write-scaling

The memcached-compatible mode is set to be compatible with the exact same way that memcache client works today.

In this mode, each GigaSpaces node runs an embedded instance of the memcached service. The communication between the memcache client and the GigaSpaces node is handled through the memcached protocol.  The GigaSpaces memcached service communicates with the GigaSpaces backend directly in-memory.

By using GigaSpaces as the backend datastore we gain immediate benefits from the robustness and richness of the GigaSpaces environment. This includes write-scaling and reliability through the built-in database integration for pre-loading data from a database and write-behind for storing data back into the database. In addition to that we gain built-in management, monitoring and deployment automation (through the GigaSpaces dev-ops API).  The fact that the entire stack is built in Java also enables to leverage the Java tool ecosystem (debugging tools, monitoring tools, profilers) as well as the portability of the Java Virtual Machine.

As the memcached protocol doesn’t include dynamic discovery, the memcached clients have static binding to the backend server, which doesn’t provide GigaSpaces’ elasticity and dynamic scaling.

Memcached load-balancer – elasticity + write-scaling, reliability

In this mode of operation, the memcached server instance does not use the embedded GigaSpaces instance but instead would reference a remote GigaSpaces cluster. In this mode, the GigaSpaces memcached server instance becomes a memcached load-balancer between the memcached clients and the GigaSpaces data partitions. Each GigaSpaces cluster could have many memcached-load-balancers. As with HTTP load-balancers, the clients only need to point to a single IP address, and every memcached request is mapped to a GigaSpaces partition according to the key used in any given operation.

The main benefit of this mode is its simplicity and elasticity. Elasticity is achieved by the fact the GigaSpaces cluster can grow as needed. The GigaSpaces proxy will discover those instances dynamically and route the memcached requests to the appropriate node. The same applies to fail-over i.e. if one of the GigaSpacs nodes fails the proxy will rote the request to a hot backup automatically. 

The memcached load balancer can be also hosted as a service within the GigaSpaces container. To gain even better scalability of the load-balancer we can combine the memcached-compatible mode (our first example)  with this load-balancer mode where each of node in the gigaspaces cluster could have the embeded memcached service configured as a load-balancer. In that configuration, a client can succeed by selecting one of the nodes in the cluster while gaining access to the entire cluster.

The downside of this approach is performance as each memcache request will to go through two network hops. For that purpose we often set GigaSpaces LocalCache to minimize that extra hop for read operations.  In this way subsequent get operation on the same key would be resolved within the load-balancer itself. Any update on that key through any of the partition will be updated automatically on the local cache as well.

Another potential use of that mode is gateway between two network domains.  It is much easier to cross firewall and network domains through a single gateway then having the entire cluster available.

Memcached RPC – Map/Reduce support, multi-lang data feeder

One of the interesting and lesser-known uses of the memcached protocol is for Remote Procedure Calls or Map/Reduce operations. In this mode, a set operation on the memcache server will be translated to a command, and a get operation is translated into a return value for that command. This is extremely useful for data-feeder scenario and aggregated query scenarios. Non-Java feeders can also leverage the GigaSpaces support for dynamic languages to pass in code that will get executed in the server close to the data.

The execution of the code would be done through the use of the GigaSpaces polling container. In this case we would set the polling container to poll for the memcache entries.

NOTE: We may be releasing a new version with built-in support for this specific pattern so the API is still subject to change ..

Simple Example:

The following example illustrates the first two modes outlined in this post – the memcached-compatible and the load-balancer modes.

For the sake of simplicity we will use a single memcached server instance. A more advanced deployment of a complete cluster is referenced at the end of this post.

Running a single memcache instance:

In this example we will run a GigaSpaces instance using the gs-memcached{.sh/.bat} . The gs-memcached utility takes one argument that points to Space URL.

> gs-memcached <Space URL>

Note for gs-memcached utility is available only since  GigaSpaces 8.0 release. For earlier versions use puInstance utility instead using the following format:

> puInstance(.sh/.bat}" -properties space embed://url=”<Space URL>” "<gigaspaces root>\deploy\templates\memcached"

Running a single memcache in compatible mode:

To run a memcached server that reference to an embedded GigaSpaces instance we will use a Space URL in the following form: “/./<name>”

In our example we will use “/./memcache” which basically set a memcache server that points to an embedded GigaSpaces instance under the name “memcached”.

>gs-memcached{.sh/.bat} ”/./MyMemcached”

or simply

>gs-memcached{.sh/.bat}

which puts a default name “/./memcached” as the Space-URL

Running a single memcached load-balancer:

To run a memcache in a load-balancer mode we would reference to remote GigaSpaces cluster by setting the Space-URL to a remote cluster as outlined below:

> gs-memcached{.sh/.bat} ”jini://*/*/remoteGigaSpaces”

A simple memcache client:

There is nothing really unique in our memcached client. We can use any standard memcached library and point it to the host:port of our server. The default port is set to11211.

   1: MemcachedClient memClient=new XMemcachedClient("localhost",11211);

   2: memClient.set("1", 3600*60, "some value");

   3: memClient.get("1");

   4: memClient.delete("1");    

Monitoring and Management:

One of the benefits of using GigaSpaces as the backend data store is the fact that we can leverage the existing monitoring and management to monitor the statistics and activity of our memcache deamon. We can use any of the GigaSpaces management tools for that purpose, including the rich client UI, the web-based UI, and command line and API based management and deployment tools.

The diagram below shows the real time statistics gathered from the gs-ui utility.

Advanced setup:

In a more advanced setup, we would want to leverage the GigaSpaces Dev-Ops API and SLA driven container to automate the deployment and managing the SLA of that deployment. A detailed description of that mode is provided in the GigaSpaces memcached documentation.

Final words – Qcon next week

We often tend to look at any new technology as like a sort of new religion. Quite often, our needs change and with that we tend to switch from one technology (religion) to another. Memcached and NoSQL are a good example for such a transition. I personally believe that to gain the benefit of a new technology we don’t need to completely abandon previous technologies but instead learn from them to make the new technologies and techniques better. That is particularly true when it comes to data technology - consider SQL, which been around for more then 4 decades!

A new writeup that came out last week on GigaOM, “Will Scalable Data Stores Make NoSQL a Non-Starter?,” is just a reminder of how quickly technology tends to shift. That makes the importance of taking a more evolutionary path toward the revolution that we’re going through even more important than ever before.

During the Qcon session Yes, SQL!  next week Uri Cohen will lay out some of the various NoSQL query languages that are available today starting with memcached, MongoDB, Cassandra, Redis and outline a model  for leveraging the best out of each in our existing JEE world. I hope it will lead to an interactive learning session and debate, as it often does at QCon. 

References

• Memcached (Wikipedia)
• MySQL And Memcached: End Of An Era? (highscalability.com)
• Did Someone Say GigaSpaces Now Has Memcached Support?  (TheServerSide.com)
• GigaSpaces memcache documentation. (GigaSpaces.com)
• GigaSpaces FREE Community edition (GigaSpaces.com)
• YeSQL: An Overview of the Various Query Semantics in the Post Only-SQL World
• GigaSpaces dev-ops API (GigaSpaces.com)
• Will Scalable Data Stores Make NoSQL a Non-Starter? (GigaOM)
• Cassandra @ Twitter: An Interview with Ryan King (twitter.com)
• Looking to the future with Cassandra (Digg.com)

NoCAP

 

In the past few months i was involved in  many of the NoSQL discussions. I must admit that i really enjoyed those discussions as it felt that we finally started to break away from the “one size fit it all” dogma and look at the data management solutions in a more pragmatic manner. That in itself sparks lots of interesting and innovative ideas that can revolutionize the entire database market such as the introduction of document model, map-reduce and new query semantics that comes with it. As with any new movement we seem to be going through the classic hype cycle. Right now it seems to me that were getting close to the peak of that hype. One of the challenges that i see when a technology reaches its peak of the hype is that people stop questioning the reason for doing things and jump on new technology just because X did that.  NoSQL is no different on that regard.

 

 

In this post i wanted to spend sometime on the CAP theorem and clarify some of the confusion that i often see when people associate CAP with scalability without fully understanding the implications that comes with it and the alternative approaches.

 

I chose to name this post NoCAP specifically to illustrate the idea that you can achieve scalability without compromising on consistency at least not at the degree that many of the disk based NoSQL implementations imposes.

 

Recap on CAP

 

Quoting the definition on wikipedia:

The CAP theorem, also known as Brewer's theorem, states that it is impossible for a distributed computer system to simultaneously provide all three of the following guarantees:^[1][2]

• Consistency (all nodes see the same data at the same time)
• Availability (node failures do not prevent survivors from continuing to operate)
• Partition Tolerance (the system continues to operate despite arbitrary message loss)

CAP and NoSQL

Many of the disk based NoSQL implementations was originated from the need to deal with write scalability. This was largely due to the changes in traffic behavior that was mainly a result of the social networking  in which most of the content is generated by the users and not by the site owner.

In a traditional database approach achieving data consistency requires synchronous write to disk and distributed transactions (known as the ACID properties).

It was clear that  the demand for write scalability would conflict with the traditional approaches for achieving consistency (synchronous write to a central disk and distributed transactions).

The solution to that was:  1) Breaking the centralized disk access through partitioning of the data into distributed nodes. 2) Achieve high availability through redundancy (replication of the data into multiple nodes) 3) Use asynchronous replication to reduce the write latency.

The assumptions behind point 3 above is going to be the center in this specific post.

 

Graphical representation of the Cap Theorem. (Source)

 

The Consistency Challenge

One of the common assumptions behind many of the NoSQL implementations is that to achieve write scalability we need to push as many operations on the write-path to a background process in order that we could minimize the time in which a user transaction is blocked on write.

The implication is that with asynchronous write we loose consistency between write and read operations i.e. read operation can return older version then that of write.

There are different algorithms that were developed to address this type of inconsistency challenges, often referred to as Eventual Consistency.

For those interested in more information on that regard i would recommend looking at Jeremiah Peschka post Consistency models in nonrelational dbs. Jeremiah provides a good (and short!) summary of the CAP theorem, Eventual Consistency model and other common principles that comes with it such as (BASE – Basically Available Soft-state Eventually, NRW, Vector clock,..).

Do we really need Eventual Consistency to achieve write scalability?

Before I'll dive into this topic i wanted to start with quick introduction to the term “Scalability” which is often used interchangeably with throughput. Quoting Steve Haines:

The terms “performance” and “scalability” are commonly used interchangeably, but the two are distinct: performance measures the speed with which a single request can be executed, while scalability measures the ability of a request to maintain its performance under increasing load

(See previous post on that regard:  The true meaning of linear scalability)

In our specific case that means that write scalability can be delivered primarily through point 1 and 2 above ( 1-Break the centralized disk access through partitioning of the data into distributed nodes. 2-Achieve high availability through redundancy and replication of the data into multiple nodes) where point 3 ( Use asynchronous replication to those replica’s to avoid the replication overhead on write) is mostly related with write throughput and latency and  not scalability. Which bring me to the point behind this post:

Eventual consistency have little or no direct impact on write scalability .

To be more specific my argument is that it is quite often enough to break our data model into partitions (a.k.a shards) and break out from the centralized disk model to achieve write scalability. In many cases we may find that we can achieve sufficient throughput and latency just by doing that.

We should consider the use of asynchronous write algorithms to optimize the write performance and latency but due to the inherited complexity that comes with it we should consider that only after we tried simpler alternative such as using db-shards, FLASH disk or memory based devices.

Achieving write throughput without compromising consistency or scalability

The diagram below illustrates one of the examples by which we could achieve write scalability and throughput without compromising on consistency.

As with the previous examples we break our data into partitions to handle our write scaling between nodes. To achieve high throughput we use in-memory storage instead of disk. As in-memory device tend to be significantly faster and concurrent then disk and since network speed is no longer a bottleneck we can achieve high throughput and low latency even when we use synchronous write to the replica.

The only place in which we’ll use asynchronous write is the write to the long-term-storage (disk).  As the user transaction doesn’t access the long-term storage directly through the read or write path, they are not exposed to the potential inconsistency between the memory storage and the long-term storage. The long-term storage can be any of the disk based alternatives starting from a standard SQL databases ending with any of the existing disk based NoSQL engines.

The other benefit behind this approach is that it is significantly simpler. Simpler not just in terms of development but simpler to maintain compared with the Eventual Consistency alternatives. In case of distributed system simplicity often correlate with reliability and deterministic behavior.

 

Final words

It is important to note that in this post i was referring mostly to the C in CAP and not CAP in its broad definition.  My points was not to say don’t use solution that are based on CAP/EventualConsistency  model but rather to say don’t jump on Eventual Consistency based solutions before you considered the implications and alternative approaches. There are potentially simpler approaches to deal with write scalability such as using database shards, or In-memory-data-grids.

As were reaching the age of Terra-Scale devices such as Cisco UCS where we can achieve huge capacity of memory, network and compute power in a single box the area’s in which we can consider to put our entire data in-memory get significantly broader as we can easily store Terra bytes of data in just few boxes. The case of Foursquare's MongoDB Outage is interesting on that regard.  10gen's CEO Dwight Merrimanargued that the entire set of data actually needs to be served completely in-memory:

For various reasons the entire DB is accessed frequently so the working set is basically its entire size Because of this, the memory requirements for this database were the total size of data in the database. If the database size exceeded the RAM on the machine, the machine would thrash, generating more I/O requests than the four disks could service.

It is a common misconception to think that putting part of the data in LRU based cache ontop of a disk based storage could yeild better performance as noted in the sanford research The Case for RAM Cloud

..even a 1% miss ratio for a DRAM cache costs a factor of 10x in performance. A caching approach makes the deceptive suggestion that “a few cache misses are OK” and lures programmers into con-figurations where system performance is poor..

In that case using pure In-Memory-Data-Grid as a front end and disk based storage as long term storage could potentially work better and with significantly lower maintenance overhead and higher determinism. The capacity of data in this specific case ( <100GB)  shouldn't be hard to fit into single UCS box or few of the EC2 boxes.

 

References

• Lessons from Pat Helland: Life Beyond Distributed Transactions

• The true meaning of linear scalability

• Consistency models in nonrelational dbs

• Foursquare's MongoDB Outage

• CAP theorem

• Why Existing Databases (RAC) are So Breakable!

• Problems with CAP, and Yahoo’s little known NoSQL system

• CAP Confusion: Problems with ‘partition tolerance’

• You Can't Sacrifice Partition Tolerance
• Gilbert and Lynch. Brewer’s conjecture and the feasibility of consistent, available, partition-tolerant web services. (2002)
• DeCandia et al. Dynamo: Amazon’s highly available key-value store. (2007)

YeSQL (part II) - Putting NoSQL, SQL and Document Model Together

In my previous post (YeSQL: An Overview of the Various Query Semantics in the Post Only-SQL World ) I introduced the common query semantics in the post Only-SQL world and made an argument that the right approach is to decouple the query semantics from the underlying NoSQL implementation. This would allow us to combine SQL semantics with NoSQL backend to achieve the best of both worlds -- standard query and scalability.

In this post I wanted to illustrate this idea through some code example using GigaSpaces as the underlying implementation of the concept. The example is based on the the same code examples from another earlier post, WTF is Elastic Data Grid? (By Example) where I covered some of the simple models for writing and reading objects to a distributed data grid. Toward the end I will reference other examples such as Datanucleus and Hbase, as well as cover patterns for supporting NoSQL semantics such as document model with existing RDBMS such as MySQL.

SQL/NoSQL Code Example (Using GigaSpaces)

The Data Model

@SpaceClass
public class Data implements Serializable {

private Long id;
private String data;
private Map info;

public Data(long id, String data , Map info ) {

this.id = id;
this.data = data;
this.info = info;

}

@SpaceRouting
@SpaceId
public Long getId() {
return id;
}

// getter/setter for id, data, info attributes omitted

}


As with the previous example, we use @SpaceRouting annotation to set the routing index.  This index determines which target partition this instance should belong to. @SpaceId determines the unique identifier (Key) for this instance.

Adding Document Model Semantics 

Most of the common examples for document-based APIs use a JSON data model as the document. A document in JSON world would look something like this:

Source: Wikipedia

From a NoSQL implementation perspective, a document often translates to a Map of Maps where each attribute is mapped to a key,value representation and nested value to a key whose value is a Map and so forth.

Document Model in GigaSpaces

In the current version of GigaSpaces a document is basically a Map attribute whose values are indexed. That gives the flexibility to add,remove attribute on an existing object without changing the object schema as with any schemaless API. A nested object would be mapped to a key whose value is a complex object that is stored just like any regular POJO.

In this example I added an info attribute that is basically a Map of Key/Values.

private Map info;


One of the main benefit of this approach is the combination of a strongly typed POJO/Table model with the flexibility of the document model. In other words, I could match objects by type, and each object can have a “fixed” structure that must conform to certain type, and a variable part that can vary on a per instance basis.  In our specific example, when I look for a Data object I'm guaranteed to get the Data.id and Data.data, but Data.info could include a variable list of attributes that can vary on a per instance basis. I can still “pin” some attributes by forcing an indexing on the relevant keys as illustrated below:

// this defines several indexes on the same info property
@SpaceIndexes( { @SpaceIndex(path = "info.address", type = SpaceIndexType.BASIC),
@SpaceIndex(path = "info.socialSecurity", type = SpaceIndexType.BASIC)
})
public Map getInfo() {
return info;
}


Note: As of GigaSpaces 8.0 release the document model semantics would be extended to support full hierarchy of Maps of Maps and dynamic indexes.

Combining SQL Query with (GigaSpaces) NoSQL Data Store

Now that we’ve gone through the API example let’s see how we could insert this data into a NoSQL data store (GigaSpaces in this specific case) and query it through SQL.

Inserting the Data Object

for (long i=0;i<1000;i++)
{
gigaSpace.write(new Data(i,"message" + i, createInfo(i)));
}


The createInfo() generates a new Map and fill it with values as can be seen below:

public static Map createInfo(long i)
{
Map< String, Object> info = new HashMap();
info.put("address", i + " Broadway");
info.put("socialSecurity", 1232287642L + i);
info.put("salary", 10000 + i);

return info;
}


Querying the NoSQL document data using SQL

There are basically two models for querying data using SQL in GigaSpaces. (1) Adding SQL-like queries using the GigaSpaces SQLQuery API (2) using a fully standard SQL jdbc driver.

Querying the data using a SQL-like model

Data[] d = gigaSpace.readMultiple(
new SQLQuery(Data.class, "info.salary < 11000 and info.salary >= 10000"),
Integer.MAX_VALUE);


The gigaspaces.readMultiple(..) operation is equivalent to a “select” statement. It takes the class (the equivalent of the from table name clause in SQL), and query clause “info.salary < 11000 and info.salary >= 10000” (the equivalent of the where clause in SQL) .

As we can see the syntax of the SQL query borrows the syntax of an object-oriented model where I can reference to the associated attributes within the document attribute just like any nested attribute. In our example info.salary would point to the info Map and pull the attribute who’s key=”salary”.

This API is useful if you're already working in POJO as your domain model, as it works with objects natively and therefore can bypass the need for any O/R Mapping.

Querying the data using standard JDBC

In this example we will illustrate how we can plug different query engines to the same data instance. In this case we will use a standard JDBC connection to connect to the data in the following way:

Connection conn;
Class.forName("com.j_spaces.jdbc.driver.GDriver").newInstance();
String url = "jdbc:gigaspaces:url:jini://*/*/myElasticDataGrid";
conn = DriverManager.getConnection(url);
Statement st = conn.createStatement();
String query = "SELECT * FROM com.gigaspaces.examples.Data";
ResultSet rs = st.executeQuery(query);
// Iterate through the result set
int i = 0;
while (rs.next()) {
System.out.println("Data [" + (i++) +"] "+ rs.getString("data") );
}


The first line sets up the GigaSpaces JDBC driver. The GigaSpaces JDBC driver is responsible for mapping the SQL query language into a the underlying GigaSpaces methods. That means that from a GigaSpaces data store perspective JDBC calls looks pretty much the same as any other call. The URL uses the following pattern <gigaspaces jdbc prefix>:<gigaspaces space url>, where the gigaspaces-jdbc-prefix is always set to jdbc:gigaspaces. The space URL points to the relevant data grid cluster. 

Note that one of the interesting concepts that comes with this is that you don’t need to point to a specific host as you would in most of the today's databases, but rather we use “*” as the host name, which initiates a network discovery using multicast to find the relevant instances of the cluster.

The rest of the code looks just like any other SQL call. We use fairly basic mapping where every class is mapped to a table and and an object attribute is mapped to a column. Note that unlike complex O/R mapping we leverage the fact that a space can store objects in their native format. That means that we don’t need to break nested objects into different tables but instead we store them as single embedded java object where the relationships are kept consistent with their original java representation.

Since the standard SQL doesn’t support nested object queries our current version of the JDBC provides access to the top level attributes.  Any nested object is treated as a POJO and is matched according to the  POJO based template matching semantics. Currently, this limits the type of queries that one can perform on nested object compared with the SQL-like API because we can’t perform matching on individual fields within the document or any nested object. As the SQLQuery and JDBC layer use the same underlying query engine, this limitation is only a semantic limitation and not a technical limitation, and will be resolved in the next release of GigaSpaces.

What About Performance?

Alex Popescu, co-founder and CTO of InfoQ.com, made a comment in his post NoSQL Databases Should Support SQL Queries questioning the performance overhead associated with adding another layer of indirection, as I pointed out in my original post. He rightfully quoted Jeff Kesselman:

The two software problems that can never be solved by adding another layer of indirection are that of providing adequate performance or minimal resource usage.

Indeed, this is one of the main challenge in this entire discussion. It is relatively simple to add another level of indirection but it's almost impossible to make it perform well.

The key to addressing the performance challenge relies on the implementation of the underlying data store and how well it is suited to support the functionality required by the higher level abstraction. In our specific NoSQL discussion the performance of the SQL abstraction would be greatly influenced by the ability of the underlying NoSQL data store to support complex queries at the core level. In this case, adding a different set of query semantics would be a matter of simple syntax mapping which should yield negligible overhead and in some cases could turn out to be more efficient, as we could support algorithms that are not simple to implement at a lower level. A good example is found in Kevin Weil's presentation Hadoop, Pig, and Twitter (slides 17-18). Kevin provides an example of how a simple Pig query could map to a fairly complex hadoop task in Java. So even if a lower level API could be more efficient at a micro level, it might turn out to be less efficient at the macro level as a result of the associated complexity. 

In our example the abstraction provided by putting Datanucleus on top of Google BigTable is probably going to yield fairly high overhead when it comes to complex queries, because most of the query semantics are implemented at the mapping layer. With GigaSpaces, we chose to support all the query semantics at the core layer and make the SQL abstraction a thin semantics mapping layer. A recent benchmark test showed only 2% difference between JDBC queries and native queries.

 

GigaSpaces Native Query engine makes the overhead 
 of the SQL abstraction negligible

Other SQL/NoSQL References

Datanucleus provides a JPA/JDO mapping layer ontop of various datastore implementations.

As such, it well positioned to provide a common SQL abstraction layer. It is also well advanced in the area of becoming a common mapping layer for multiple NoSQL datastores. Currently it supports HBase, BigTable, Amazon S3. MongoDB and Cassandra are currently works in progress.

As there is still a big difference between the various datasource implementations (each supports only a subset of query semantics), each datasource  plugin  covers a subset of Datanucleus features. There are various documented limitations per datastore. You can find the list of supported features in the relevant datastore plugin documentation (a reference to googlestorage is provided here ).

You can also find a useful code example for using the Datanuclues JPA plugin for Hadoop/Hbase in the following post Apache Hadoop HBase plays nice with JPA by  Matthias Wessendorf.

RDBMS Support for NoSQL Semantics

As I was working on this write-up, which centers around supporting SQL on top of a NoSQL datastore, I came across an interesting post: Schema-Free MySQL vs NoSQL by Ilya Grigorik, CTO/Founder at PostRank.

In his post, Ilya provides an interesting pattern that shows how you can store a schemaless document model on a MySQL datastore.

Instead of defining columns on a table, each attribute has its own table (new tables are created on the fly), which means that we can add and remove attributes at will. In turn, performing a select simply means joining all of the tables on that individual key

Ilya ends with a comment that I thought was in line with the my thoughts outlined in this post:

..there is absolutely no reason why we can’t have many of the benefits of “NoSQL” within MySQL itself.

Final Words

In this post I've tried to illustrate, mainly through the GigaSpaces example, how the SQL and NoSQL models can be brought together to achieve the best of both worlds, i.e., the scalability of the NoSQL model and the rich and standard query semantics of SQL. At the same time, I believe that additional semantics that are not currently supported by SQL such as the document model and object relationship can also be brought together as an extension to current SQL semantics as illustrated in this post.

Having said that, there will be a time where a Object or document-centric language are more intuitive or simpler to use then SQL. The decoupling of the datastore and the query language should enable us to write objects in an Object/Document-centric model and still query it using a SQL engine and vice versa.  This will give us more flexibility to choose the language that best fits the context in which it is being used, and mix and match among them just as we do today in any web application where we mix together different languages such as  HTML, CSS, JavaScript and other languages.

Finally, I believe that large part of the discussion in the NoSQL community took a wrong turn by putting too much emphasize on the query language rather then the scalability patterns which IMO should remain the only motivation for switching between one datastore to the other.

References

• YeSQL: An Overview of the Various Query Semantics in the Post Only-SQL World
• WTF is Elastic Data Grid? (By Example)
• Apache Hadoop HBase plays nice with JPA by  Matthias Wessendorf
• Hadoop, Pig, and  Twitter presentation by Kevin Weil 
• Schema-Free MySQL vs NoSQL by Ilya Grigorik
• NoSQL databases Should Support SQL Queries by Alex Popescu 

YeSQL: An Overview of the Various Query Semantics in the Post Only-SQL World

The NoSQL movement faults the SQL query language as the source of many of the scalability issues that we face today with traditional database approach.

I think that the main reason so many people have come to see SQL as the source of all evil is the fact that, traditionally, the query language was burned into the database implementation. So by saying NoSQL you basically say "No" to the traditional non-scalable RDBMS implementations.

This view has brought on a flood of alternative query languages, each aiming to solve a different aspect that is missing in the traditional SQL query approach, such as a document model, or that provides a simpler approach, such as Key/Value query.

Most of the people I speak with seem fairly confused on this subject, and tend to use query semantics and architecture interchangeably. So I thought that a good start would be to provide a quick overview of what each query term stands for in the context of the NoSQL world. Then, I'll try to break some common misconceptions -- which led me to come up with the YeSQL term.

Common Query Semantics in the Post Only-SQL world

The following are some of the common query semantics in the NoSQL world. For those that are interested in code examples i’ve linked each category with the relevant GigaSpaces reference API.

• Key/Value query: Key/Value query, as the name suggests, is probably the most basic form of query. Each data item is associated with a unique identifier (key). In the NoSQL world, memcache is one of the most common implementations of such an interface. A common pattern to perform complex queries with memcache is to defer them to an underlying database which is used as a search engine. The result of these queries is a key or set of keys that is then used to perform subsequent fetching of the values through the memcache data store. The reason they gained new momentum in the post-SQL world is because they lend themselves fairly natively to the concept of partitioning and distribution, which is a key piece in making a data store scalable. In other words, people were willing to trade the rich query functionality provided by most traditional RDBMS for scalability with only basic query support if that was their only choice.

• Document-based query: The roots of this model are in the search engine world, where it is very common to store different types of documents, even if each one represents a completely different object. In the NoSQL world, a document is not the typical Word or PowerPoint that you would see in search engine, but rather objects in the form of Jason or XML, or binary objects associated with a set of key/values, as in the case of Cassandra. In SQL terms, a document can be seen as a blob that is associated with a set of keys, each indexed independently and maintaining a reference to this blob. Each blob can be a different type (tables), each blob can have different set of associated indexes (keys). Matching is done through the associated indexes. The result-set often includes multiple types, each containing a different set of data. Because the indexes and the blob don’t need to conform to a strict structure of rows and tables, it is referred to as “schemaless” -- i.e., it can have different versions of the same type, and add fields to new types without having to modify any table or update older version of the data. Examples that support the document model are CouchDB and MongoDB.

• Template query: Template queries were common in JavaSpaces and even in later versions of Hibernate. With template-based matching, you can fetch an object based on class type or inheritance hierarchy, as well as values of the attributes of that object. In more object-oriented versions of template matching you can also perform matching based on specific items within a graph attribute. GigaSpaces is one of the better-known implementations that support the JavaSpaces template query model.

• Map/Reduce: Map/Reduce is often used to perform aggregated queries on a distributed data store. A simple scenario would be Max, or Sum. In such scenario the query request need to be executed independently in each partition (Map) and then aggregated back to the client (Reduce). An implicit Map/Reduce can take a certain query request and spread the execution of that query implicitly. The client gets the aggregated query as if it was a single query. The explicit model allows you to execute code in free-form, where you can control the mapping model -- which call goes to which data, the code to run in each node (aka tasks), and the results. In a typical Hadoop implementation, Map/Reduce is often done through the explicit model. Frameworks like Hive and Pig provide an abstraction model that can handle that process implicitly.

• SQL query: If you think about it, SQL is yet another form of dynamic language that was specifically designed for complex data management. With SQL, data is often ordered in tables and rows. Some of the query semantics in SQL, such as Joins, distributed transactions, and others are known to be anti-pattern for scalability. It is mostly this aspect of SQL semantics that associates SQL with scalability limitation. Examples of NoSQL implementations that support SQL are Google Bigtable using JPA, Hive/Hadoop, MongoDB and GigaSpaces. I will discuss in further details below what that actually means. 

YeSQL -- There's Nothing Wrong with SQL!

Now that we've covered some of the concepts behind query formats, it becomes more apparent that there is nothing really wrong with SQL. Like many languages, SQL gives you a fairly long rope with which to hang yourself if you choose to, but that is true of almost any language. If you design your data model to fit into a distributed model, you may find that SQL can be a fairly useful format to manage your data. A good example is Hive/Pig/Hbase and Google JPA /Bigtable. In both cases the underlying data store is based on a scalable Key/Value store, but the front-end query language happens to be SQL-based. MongoDB aims toward a similar goal with the main difference that it provides SQL-like support and doesn’t fully comply with any of the existing standards.

Its About the Architecture, Stupid!

NoSQL implementations such as Hive/HBase as well as JPA/Bigtable can be a good example of how next-generation databases can support both linear scaling and a SQL API.

The key is the decoupling of the query semantics from the underlying data-store as illustrated in the diagram below:

Supporting a SQL API on top of a NoSQL data store in Google Bigtable

Convergence is Underway

Last week I spent some time at the Hadoop summit. Hadoop created a fairly generic substrate that led to an innovative ecosystem behind it. There are already many new frameworks today that provide different levels of abstraction to the way Hadoop manages data in both query and processing, such as Hive, Cascading, and Pig. Many of them provide tools that the original creator of Hadoop never even thought of.

Which brings me to the point that we can apply the same decoupling pattern that I mentioned above to support a document model in connection with SQL. In other words, I believe that going forward, most of the leading databases will support all of the semantics listed above, and we won't have to choose a database implementation just because it support a certain query language.

We've already seen a similar trend with dynamic languages. In the past, a language had to come with a full stack of tools, compiler, libraries, and development tools behind it, making the selection of a particular language quite strategic. Today, a JVM in Java or a CLR in .Net provides a common substrate that can support a large variety of dynamic languages on top of the same JVM runtime. Good examples are Groovy and Java or Jruby.

Final Words

As I pointed out throughout this post, SQL is actually a fairly good query language and will continue to serve a major role in the post only-SQL world. However, the concept of one size fits all doesn't hold up. The data management world is going to be built from a variety of tools and data management languages, each serving a particular purpose. Ideally, we should be able to access any data using any of several query languages, regardless of how it was stored. For example, I should be able store a Jason object using a document model and then, at any time, query that Jason object using SQL query semantics or a simple Key/Value API.

 

References

• YeSQL (part II) - Putting NoSQL, SQL and Document Model Together
• NoSQL and NewSQL (Presentation)
• Integrating Hive and HBase
• Apache Hadoop HBase plays nice with JPA
• Using JPA with App Engine
• GigaSpaces Key value query, Document based query, Template query , Map/Reduce, SQL query
• Query Processing for NOSQL DB

Memory is the New Disk for the Enterprise

Back in June 2008 i wrote a post in response to an InfoQ article:  RAM is the new disk... where i quoted Tim Bray and others comparing file system performance to memory:

Memory is the new disk! With disk speeds growing very slowly and memory chip capacities growing exponentially, in-memory software architectures offer the prospect of orders-of-magnitude improvements in the performance of all kinds of data-intensive applications. Small (1U, 2U) rack-mounted servers with a terabyte or more or memory will be available soon, and will change how we think about the balance between memory and disk in server architectures....

… Disk will become the new tape, and will be used in the same way, as a sequential storage medium (streaming from disk is reasonably fast) rather than as a random-access medium (very slow). Tons of opportunities there to develop new products that can offer 10x-100x performance improvements over the existing ones.

It didn't take long for this vision to become a reality through the work that we're doing these days with Cisco. By integrating GigaSpaces XAP with the Cisco UCS machine we are demonstrating our ability to easily load hundreds of gigabytes into a single box, and to scale linearly with growing capacity without any performance degradation. This is a great example of how middleware that was built for memory from the ground up, combined with hardware that was equipped to provide terabytes of memory in a single box, can be game changing.

This exciting combination makes it possible to manage 15-20x the amount of data in-memory, per partition. This, in turn, makes it possible to store the entire application data set in‑memory, and gain not only 10x the performance but also great simplicity, because the application no longer needs to deal with a *miss* ratio in the cache; and, at the same time, there are no consistency issues because all the data resides in-memory.

The table below was taken from a recent Stanford research paper -- The Case for RAMClouds. It provides estimates of the total storage capacity needed for one-year’s-worth of customer data for a hypothetical online retailer and a hypothetical airline. In each case the total requirements are no more than a few terabytes, which would fit in a modest-sized RAMCloud. The last line provides the estimated cost for purchasing the necessary RAMCloud servers :

As the table illustrates, storing the entire annual data of an online reservations system or an online retailer can be mapped completely into memory. The combination with UCS makes the deployment of such a system simple to manage and economic. The online retailer's annual data can fit into 4 UCS chassis, and the entire annual data set of the airline reservation system can fit into 2 UCS chassis!

The Next Big Thing -– In-Memory Elastic Database

In one of my recent talks, Elastic Data on the Cloud: Hype or Reality? I also discussed the trend in the context of the cloud database. I spoke about how the addition of elastic middleware  -- which enables allocation of data service on-demand, sharing that data with other tenants of the application, and provides built-in dynamic scaling as needed -- will make this sort of deployment significantly simpler and more economical.

  

What’s Next

In my last CTO Note I described our current progress with our 7.1 XAP release and our plans toward the future 8.x release:

Here is what we’ve done to take XAP even closer to this vision, in the recent 7.0 release and the upcoming 7.1 release (slated for April 14, 2010):

• Fast network –- XAP 7.1 will make dynamic scaling simple by enabling you to set up distributed system with a single API call. This not only provides a simpler API for developers, but also eliminates manual work for sizing, provisioning, and configuration at the deployment stage, and reduces the maintenance effort by automatically adjusting cluster configuration in response to changing loads, machine failure, and other events.
• Large memory -– XAP can manage terabytes of data in-memory. It can even manage up to 384 GB in a single VM, through its certification for use with Cisco UCS, coming up in version 7.1.
• Multi-core -– as of version 7.1, XAP comes with new concurrent transaction management for better utilization of multi-core processors. This comes on the heels of major improvements in multi-core utilization in version 7.0, which resulted in a 300% performance improvement on the Intel Nehalem processor. New benchmarks will shortly be released showcasing the performance of XAP 7.1 on Cisco UCS, which is based on the latest multi-core technology.
• Virtualization and multi-tenancy –- XAP 7.1 is the first middleware platform with built-in multi-tenancy, making it extremely simple to share applications on the same resources with full isolation and enterprise-grade security. This dramatically cuts the cost per user for software vendors that use XAP for SaaS-enablement, because it enables them to cram more users onto the same hardware resources. Full VMware integration is also available, enabling multi-tenancy throughout the entire virtualized application stack.

Our 8.x release will include more exciting stuff in the same direction, with JPA support and continuation of our work to make our elastic middleware simpler. We will also be releasing a tighter packaging with Cisco UCS that will make large data deployments fully optimized from the application down to the hardware.

References:

• InfoQ Article - RAM is the new disk...

• Why Existing Databases (RAC) are So Breakable!

• The Common Principles Behind the NOSQL Alternatives

• The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM By Department of Computer Science Stanford University

• Are Cloud Based Memory Architectures the Next Big Thing?, highscalability.com

• The Missing Piece in the Virtualization Stack (Part 2)

Multi-tenancy: does it have to be that hard?

Multi-tenancy is a term that is used often in the context of SaaS applications. In general, multi-tenancy refers to the ability to run multiple users of an application on a shared infrastructure. The main motivation for doing this is efficiency, or in other words -- reducing the cost per user in comparison to a dedicated system where each user has their own dedicated environment.

Over the past few months I've had discussions with various SaaS providers, in which I tried to learn about the main challenges that SaaS providers face when they deal with multi-tenancy. As it happens, I came across an interesting article on this very topic, Why Multi-tenancy Matters in the Cloud by Alok Misra. The article later sparked an interesting discussion thread on the cloud mailing list, which I also find to be a good reference for diverse thoughts on this subject.

In this post I want to summarize some of my findings on multi-tenancy, as well as suggest a model that can make multi-tenancy significantly simpler.

The common challenge: Multi-tenancy at the data layer

There are many approaches and levels of multi-tenancy, depending on the application layer and type of application you're dealing with. For example, trying to separate user accounts in a CRM application is quite different than separating various batch processing jobs in a risk management application. That said, what seems a common challenge in almost every SaaS application is multi-tenancy at the data layer. To put it simply, the challenge is how to share data resources (database in most cases) among multiple users, while at the same time ensuring data isolation between those users, as if they are running on completely physically seperated servers.

In the remaining part of this post I'll address how multi-tenancy is currently being dealt with at the data layer, and how the existing challenges can be met.

How is multi-tenancy being done today?

For this part of the story, I think it might be useful to first look at the two extreme approaches: The fine-grained approach and the cross-grained approach.

Fine-grain multi-tenancy (Salesforce.com model)

For many SaaS providers the “poster child” of multi-tenancy is Salesforce.com. There is a very useful videocast explaining the Salesforce approach to multi-tenancy. In a nutshell, their approach is to share the same database instance between different users, and define the data model so that each table has a primary key identifying the customer ID. In addition, there is a separate metadata table that translates every query into a user-specific query before the query hits the database.

• Pros:
  • Very fine-grained multi-tenancy resulting in optimum cost efficiency per user.
• Cons
  • Complexity: Requires changes in the data model.
  • Doesn’t fit to existing application: Existing applications require a complete rewrite.
  • Modest degree of isolation: Users sharing the same resource can impact one another under heavy load situations. A maintenance/patch release required for a particular user might affect other users. In a more extreme scenario, there is also exposure to a higher degree of vulnerability. For example, a bug in the query mapping could point one user to data belonging to other users just by picking up the wrong customer ID.

Cross-grain multi-tenancy (hosted service model)

Most hosting providers (including hosted IT providers) enable outsourcing of certain IT applications. Efficiency is gained by sharing maintenance overhead, labor costs, and other data center costs such as network, electricity, and more.

• Pros:
  • No code changes are required, resulting in greater simplicity.
  • High degree of isolation.
• Cons
  • Lower efficiency and therefore limited cost margins.

Challenges with today's approach to multi-tenancy

The challenge with the fine-grained model is that it is fairly complex to maintain and implement. If you have an existing application, it requires a complete rewrite and also forces fairly significant changes in your existing data model. The challenge with the cross-grained model is that cost margins per user are still rather high. In a competitive environment, even a single competitor offering similar servicse to yours using the fine-grained approach could easily put you out of business.

The elasticity challenge

The other challenge that neither approaches addresses today is dynamic elasticity. What happens when a customer grows beyond their allocated shared capacity? Both approaches seem to rely on certain capacity assumptions, and can scale by splitting multiple users between their shared resources. However, none of the approaches seem to handle a situation where a particular user's capacity grows beyond the allocated resources.

Jayarama Shenoy provides a good introduction to that challenge in his comment in the discussion thread:

So, how does elasticity work in a multi-tenancy case? Say 100 tenants share an app on a server (by which I would think they go all the way down to a shared database). Then,say, five of them grow quite rapidly such that they need to be moved off that server, and the other 95 can continue to exist on the original server.
So 'something' in the application has to know how to pick off the records that belong to these five, copy it off into a second instance of that database and kill, move & restore application service for those tenants. (Of course, without being privy to the data contained in those records). And of course there's an implied load balancer element in front of all this to keep up appearances to the clients.

Based on this analysis I believe that the main challenge with the current approaches is that we have too few options available. No option seems to provide a good solution to the elasticity challenge, and this is what is making multi-tenancy “so damn hard”. This is especially true for existing businesses with existing products.

The solution: Making multitenancy simple

Learning from the past operating system experience

Some of us probably remember the days when Bill Gates came up with DOS, which was basically a single-user system. At the time, if you wanted to provide a multi-user application you had to write it yourself, and sometimes that required going down to the core interrupt level and program in assembly language. This made multi-user systems on DOS very uncommon. Unix, on the hand, was built for multitasking, multi-user environments, and therefore it was quite common to build multi-user systems in the Unix environment. Later on, when Windows NT came out, all that changed. Today we can’t even imagine a scenario where we would build an application that wouldn’t support multiple users. 

Lesson learned:

As long as the underlying infrastructure (the operating system in the example, and the database in our specific discussion) is not built for multi-tenancy, trying to solve the multi-tenancy challenge at the application level is doomed to be as complex as writing a multi-user system on DOS. 

In fact, once multi-tenancy was dealt with properly at the operating system level, writing a multi-tenant application became significantly simpler and therefore widely adopted. The same should apply to databases as well.

The database is the problem!

Almost all of the attempts I've seen in both the Salesforce approach and the hosted environment approach were built under the assumption that the database is a fixed “concrete” construct that cannnot be broken down easily into “smaller bricks”, let alone scale dynamically. It is therefore not surprising that most of the attempts to deal with multi-tenancy have tried to address the challenge outside the database.

Now, imagine what your approach to the challenge would be if a database could be easily broken down into small “bricks” without huge overhead, and if it could scale dynamically?

Multi-tenant data service

Similar to the experience we had with operating system, I believe that the solution for multi-tenancy has to be dealt with at the infrastructure level. In our specific discussion, that would be the database. In this section I’d like to draw the principles for a multi-tenant data service:

• Simplicity: With a multi-tenant database, applications can be written to a single database tenant just as they are today. How tenants are allocated and shared with other tenant needs would be completely abstracted from the application.
• Efficiency: Efficiency is maintained by the fact that all data tenants can share the same hardware (memory, CPU) with other tenants.
• Dynamic scaling: Each database tenant can be distributed and scale dynamically across multiple machines, if necessary, to meet demand. This includes moving data to other machines if necessary. Scaling must be supported seamlessly without any downtime.
• Support for multiple isolation/sharing levels: With a multi-tenant database, the level of sharing/isolation is set on demand, based on the specific requirements. At that level, a user can choose a dedicated tenant, in which case the multi-tenant database would allocate tenants on a set of machines that is dedicated to that specific user. Clearly, the benefit of this is isolation, at the cost of efficiency. Another type of sharing is available to the user at the same time: Choosing to share with any individual or group under the user's control. In other words, rather then dictating either extreme (full isolation or full sharing) for all users, with a multi-tenant database we can dynamically choose the right level that best fits our purpose.

Where we are today?

Cloud providers such as Amazon, Microsoft, and Google have already realized that they can’t expect their users to deal with multi-tenancy themselves. So they've come up with data services that were designed with multi-tenancy in mind. Amazon recently introduced Amazon RDS (Relational Database as a Service) and Microsoft introduced SQL Azure. Both are an attempt to take existing databases (MySQL in the case of Amazon and SQL Server in the case of Microsoft) and add some of the characteristics that I discussed above. For example In both cases the user works with a single tenant and is completely abstracted from the underlying resource sharing optimizations. 

And yet, both solutions are still fairy limited when it comes to providing elasticity: With the Amazon solution, you're expected to scale-up (use a bigger machine) and go through a period of downtime when you switch between one degree of scaling and another. Google AppEngine has taken a different approach. Instead of relying on an existing RDBMS and wrapping it up with multi-tenancy and dynamic scaling, they took their own NoSQL alternative (BigTable) that was already built for dynamic scaling and, through a mapping layer, provided SQL semantics on top of it. As with RDS and SQL Azure, the user is completely abstracted from the way Google manages multi-tenancy in their underlying infrastructure.

Niether Amazon, Microsoft, nor Google provide WRT flexibility to the isolation level, and more importantly, all these solutions seem to be completely tied into their own cloud data center, which means that I cannot port it to my own local environment. From a throughput efficiency perspective, SQL Azure and Google seem to trade scaling for throughput by introducing an additional layer of indirection (load balancer in the case of Azure) that makes their multi-tenant solution fit only low-end applications.

At GigaSpaces we took an approach very similar to the Google approach with our new Elastic Middleware Services, where we use our existing distributed NoSQL In-Memory Data Grid and used it as a muti-tenant data service.

That is it for similarity. Here are the differences:

• You can easily download and install GigaSpaces in your local environment or use it in the cloud and get the full multi-tenancy and dynamic scaling experience. We provide an open SPI that enables you to plug in any resource pool, such as starting from a simple SSH call through a complete virtualization pool.
• You can control the level of isolation, scaling, and other SLA directly through an API.
• Our data services are designed for extreme efficiency by maximizing the utilization of memory resources. This normally translates into the ability to run at least 10x more user transactions on the same hardware, which in turn translates into huge cost savings.
• We provide the same consistent approach for other layers of the stack such as messaging and map/reduce through the same environment.

You can learn more about the Elastic Middleware solution in my previous post.

References

• SaaS Enablement of existing applications (Videocast)

• The Missing Piece in the Virtualization Stack (Part 2)

• Why Multitenancy Matters In The Cloud by Alok Misra

• Multi Tenancy - Keep It Simple, REALLY Simple by Ayende Rahien
• Multitenancy definition – Wikipedia
• Elastic Data On the Cloud: Hype or Reality 
• GigaSpaces 7.1 Release
• Multitenant Magic - Under the Covers of the Force.com Data videocast by Craig Weisseman Chief Software Architect salesforce.com
• Security and multitenancy with GigaSpaces XAP (docs)
• Security and multitenancy with GigaSpaces XAP (videocast)

The Missing Piece in the Virtualization Stack (Part 2)

In the first part of this post, I discussed how virtualization and cloud computing, as we know it today, is only a small part of the solution for today’s IT inefficiencies. While new technologies and delivery models have made it much simpler to manage the infrastructure, this is not where our core inefficiencies lie. Virtualization principles must be extended to higher levels of the application stack, to make it easier for all of us to manage, tune and integrate applications. Otherwise we will continue to spend most of our time on things that don’t provide real value to the business – infrastructure, installation, management, tuning, and integration.

In this post, as promised, I’ll show how this missing piece can be filled in practice, by something called Elastic Middleware Services. These are middleware services that can be deployed with one API call, and are completely abstracted from the applications that use them. Elastic Middleware Services are very similar to cloud-based middleware like Amazon SimpleDB, but the major difference is that they are enterprise-grade, and can be purchased and installed in your private data center, for use by your internal enterprise applications.

To better illustrate this idea, I’ll discuss a reference implementation of Elastic Middleware Services which will shortly be released as part of GigaSpaces XAP 7.1. The discussion is intended to illustrate in concrete terms how to fill the missing piece in the virtualization stack. I’ve intentionally focused on the conceptual and fundamental elements of the GigaSpaces implementation, not on specific features and benefits, to make the discussion useful even if you are not using or considering GigaSpaces products.

By the way, I recently hosted a webinar on this topic, together with Uri Cohen, GigaSpaces’ product manager. In the webinar we presented GigaSpaces’ upcoming Elastic Middleware Services. The recorded version of this presentation is available below:

Elastic Middleware Services

The idea is to provide middleware services (messaging, data, Map/Reduce, and more), like those the ones provided by cloud providers such as Amazon and Microsoft Azure, in your own local environment. These services simply extend the basic principles of virtualization, as I outlined in the previous post, to the middleware stack:

Basic virtualization principles

1. Break big physical resources into smaller logical units
2. Decouple the application from the physical resources
3. Provide an abstraction that makes all the small units look like one big unit

From a middleware perspective, this means that instead of having one big database, you have lots of small units of data services (also called partitions or shards), which are grouped together through a client side proxy that exposes all of these small units as one big unit to the client that is using them. The client proxy is also responsible for abstracting the client from the physical location of those resources. To scale, all you need to do is add more of those units. The client experiences these additional units as extra capacity added to the service.

Deploying middleware with one API call

From an API perspective, this is a much simpler approach than traditional middleware, because it introduces a higher-level abstraction to the user, as can be seen in the code snippet below:

With this new elastic middleware API, users define requirements in a language that is closer to their domain. For example, “I want to create a data service that will have 10GB of data and can grow up to 100GB of data, I need hot failover and I'm willing to share my deployment with other users of my own organization but not with users of other organizations.”

In response, the Elastic Middleware Services would automatically do the following:

• Allocate the right number of partitions, based on machine availability and the available memory in each machine, to address the capacity required.
• Automatically launch a hot backup in for each partition and ensure that it runs on separate machine than the primary partition, to address the high-availability requirement. 
• Make sure that if another organization’s application is already running on the machine, the application won’t deploy on that machine, to address the security requirement.

Fully automated through built-in SLA

As I noted in my previous post, one of the major costs of enterprise applications over a period of three years is the cost of maintenance and operations. The main component of maintenance costs is the labor cost, closely tied to the amount of manual work you have to put into your system to make sure that it meets the application’s SLA. The cost grows proportionally to the demand for scale.

With Elastic Middleware, maintenance cost can be substantially reduced through built-in SLA and automation that covers the following aspects:

• Scaling SLA – scale when there is a memory shortage, high CPU utilization, Garbage Collector hiccups, etc. Users can choose to scale automatically when any of these events occurs, and customize their own thresholds (e.g. scale when CPU utilization hits 90%). The Elastic Middleware is integrated with a cloud or virtualization framework, to enable it to automatically pull or release of machines as needed.
• Failover – when a machine fails, the Elastic Middleware does one of two things:
  • Scale down – if there are available resources to meet the current workload, the system will automatically scale down and continue to service the application.
  • Rebalance – if there are not enough resources to serve the application, the Elastic Middleware calls the cloud/virtualization pool and starts a new machine. As soon as the new machine starts, it is added to the existing pool of resources, and the application is re-balanced to take advantage of the additional capacity available through the new machine. It is important to point out that if you don’t have a dynamic pool of resources available through virtualization or cloud computing, you could still start a machine manually. Once the machine is started, adding it to the application’s resources and rebalancing would happen in exactly the same way.
• Continuous rebalancing – when new machines are added to the Elastic Middleware, it immediately detect them and re-balances the assets currently running on the new machines (if necessary), ensuring optimum utilization of all services.

Carrying out this dynamic SLA was deigned to happen while the application is running, ensuring no transaction or data loss during the process. In-flight transactions continue to be served with no noticeable hick-ups. 

Concerned of losing control?

Whenever I introduce the concept of built-in SLA and automation, I get two type of responses. One is “Wow, that’s cool!”, the other is “It sounds like I’ll lose control.” The concern about losing control is very valid, as in many cases, especially in non-cloud environments the concept of spare capacity rarely exist and therefore adding a new machine would often involve some manual intervention. In addition, when something goes wrong, other parts of the system or even other parts of the organization need to be involved, so we can’t always assume that full automation is possible.

The cruise control analogy

The approach GigaSpaces has taken is similar to the way cruise control works in our cars. You can choose to give up full control under a certain threshold, but at each point in time you can take the wheel back and resume full control. Like many of the cloud providers, we provides this type of cruise control for our services through an open API that enables management and control of every aspect of our cluster. You can query the available node, CPU, application partitions and service components. GigaSpaces Elastic Middleware was built as a layer on top of that API, which means that if the built-in functionality that comes with the Elastic Middleware doesn’t fit your needs, you can go one layer down and write your own custom behavior. For example, you can specify that in your system, scaling or failover events send an alert that will trigger a manual process for adding a new machine. In other words, you have full flexibility to choose when to turn on “cruise control” and when to take ownership.

SaaS-enabled with built-in multi-tenancy

Multi-tenancy often means that can run multiple users/customer applications on a shared resource, thereby reducing the cost of ownership per user. This is considered an area of high complexity, specifically in SaaS applications, because with today’s middleware, the application itself is responsible for mapping between users and application tenants. With Elastic Middleware Services, multi-tenancy is built into our implementation and API. Thus the burden of sharing resources is moved from the application down to the middleware. Each application works with its own dedicated middleware service (data, messaging, etc.) but that middleware service can share resources with other middleware services. In other words, instead of having one big database running on a dedicated machine and split it out between users at the application level, you can have lots of small databases spread between machines, where each application can have its own dedicated database and still have that database shared with other database instances that runs on that same machine.

This not only significantly simplifies your code, it also provides better isolation between multiple users, as well as independent life cycle management for each tenant.

There are various tradeoffs between sharing and isolation. With isolation you get better security and control of your own environment, but lower utilization and higher cost. With sharing, you can reduce cost and still achieve reasonable isolation, but not at the same level as when running on a dedicated resource. The Elastic Middleware Services make it possible to define the isolation/sharing at the application level. It does this by introducing the notion of an isolation level:

Currently we support three isolation levels:

• Dedicated – guarantees a dedicated machine allocated per instance of the application.
• Shared private – multiple instances of the application or organization share the same resources, but other departments or organizations are isolated.
• Shared public – everyone shares everything.

 

Available both on your local network and on the cloud

To try the new service you don’t need to have a private cloud or run your application on a public cloud. You don’t even need a virtualization layer. All you need is to launch a single GigaSpaces agent (Java process) per machine (normally that process will be started automatically at machine boot).

Once you’ve done this, you can start interacting with your machines and an create the desired middleware service through a simple API call. The following snippet shows how you might use this API:

Admin admin = new AdminFactory().createAdmin();

ElasticServiceManager elasticServiceManager = admin.getElasticServiceManagers().waitForAtLeastOne();

// Start a new data-grid

ProcessingUnit pu = elasticServiceManager.deploy(new ElasticDataGridDeployment("mygrid") // give it a name

.isolationLevel(IsolationLevel.DEDICATED) //isolation level

.highlyAvailable(true) // set the high availability level

.elasticity("2GB", "6GB") // set the required capacity range

.jvmSize("512MB") // configure the size per VM

.addSla(new MemorySla(70))); // define Memory SLA

That’s it! – you just got a full cluster deployed and ready to use.

What does it mean for you?

To sum up, I here are the main benefits that Elastic Middleware Services bring to each type of user:

For developers

As a developer, you can get access to the service you want just by calling an API, without worrying about installation or cluster setup. You get access to the service from your existing platform. In other words, you don’t need to run your application on GigaSpaces XAP to use any of those services. You can also pick and choose – use only the services you want, e.g. just the data, data and messaging, etc.

For the (private) data center

From a data-center perspective, you could install Elastic Middleware Services only once in your data-center. The same installation and resources would be shared amongst all users in your organization. If you happen to have virtualization in place, you can do this very easily using the GigaSpaces virtual appliances. 

Once you install the system, other users in the organization don’t need to go through the installation process – they can start consuming the middleware services just by calling an API, like they would do on the Amazon cloud when they launch a SimpleDB instance.

In a way, the Elastic Middleware Services gives the data center the power to provide higher-lever services to the business, rather than just plain infrastructure services, enabling the organization to become more agile.

For public cloud providers

Public cloud providers can be viewed as an outsourced version of the local data center, so they should experience much the same benefits as data centers. In addition, with Elastic Middlware Services, public cloud providers can offer a set of middleware services that are enterprise-ready with extremely high utilization per service. Because the Elastic Middleware Services are fully memory-based, you can use as much as 10X less machines to achieve the same throughput requirements, compared to a disk-based approach.

Ricky Ho published some interesting performance characteristics of Amazon SQS and SimpleDB in one of his recent posts:

• Network latency and throughput: 20 - 100 ms for SQS access, SimpleDB domain write throughput is 30 - 40 items/sec.
• Eventual consistency: 2 simultaneous requests to dequeue from SQS can both get the same message. SQS sometimes reports empty when there are still messages in the queue

While both SQS and SimpleDB provides a scalable data store and messaging services, their performance are at least 10-50X slower than an equivalent GigaSpaces data and messaging service. Implementing a Map/Reduce scenario with GigaSpaces is going to be significantly simpler and closer to real-time compared to Amazon Elastic Map Reduce (and that’s a topic for a separate post). Another cool feature is the ability to execute your code within the service container, which gives you an even bigger improvement in performance and latency.

The Elastic Middleware Services have much higher utilization and simpler maintenance, compared to equivalent services offered separately to the end-user, because they are provided using a shared cluster with a unified clustering model. With Elastic Middleware Services you get JPA, memcached, key/value (NoSQL), Spring, JMS, Remoting and Map/Reduce in Java or .Net, in a single deployment.

Existing GigaSpaces XAP users

Existing GigaSpaces XAP users can use the Elastic Middleware Services as part of XAP.  They’ll benefit from the simplicity provided through the high-level abstraction. In addition, the Elastic Middleware Services make it easier to plug-in the GigaSpaces components into existing application servers or development environments. They also make it significantly simpler to manage and deploy GigaSpaces XAP in various groups in the organization, because they can all share the same virtual pool of machines in the data center.

Availability

The Elastic Middleware Services will be available as part of our upcoming XAP 7.1 release, due for end of March 2010. It is currently available for private beta testing. We will also release a public beta of the Elastic Middleware Services through our upcoming 7.1 release candidate version. As always, we’ll welcome feedback and will be very happy to hear about your specific requirements. You could either post comment to this post or send an email to <pm at gigaspaces dot com>.

The Common Principles Behind the NOSQL Alternatives

A few weeks ago, I wrote a post describing the drive behind the demand for a new form of database alternatives, often referred to as NOSQL. A few weeks ago during my Qcon presentation, I went through the patterns of building a scalable twitter application, and obviously one of the interesting challenges that we discussed is the database scalability challenge. To answer that question I tried to draw the common pattern behind the various NOSQL alternatives, and show how they address the database scalability challenge. In this post I'll try to outline these common principles. 

The Common Principles Behind the NOSQL Alternatives

Assume that Failure is Inevitable

Unlike the current approach where we try to prevent failure from happening through expensive HW, NOSQL alternatives were built with the assumption that disks, machines, and networks fail. We need to assume that we can’t prevent these failures, and instead, design our system to cope with these failures even under extreme scenarios.  Amazon S3 is a good example in that regard. You can find a more detailed description in my recent post Why Existing Databases (RAC) are So Breakable!.  There I outlined some of the lessons on how to architect for failures, from Jason McHugh's presentation. (Jason is a senior engineer at Amazon who works on S3.)

Partition the Data

By partitioning the data, we minimize the impact of a failure, and we distribute the load for both write and read operations. If only one node fails, the data belonging to that node is impacted, but not the entire data store. 

Keep Multiple Replicas of the Same Data

Most of the NOSQL implementations rely on hot-backup copies of the data, to ensure continuous high availability.  Some of the implementations  provide you with a way to control it at the API level, i.e. when you store an object, you can specify how many copies of that data you want to maintain at the granularity of an object level. With GigaSpaces, we are also able to fork a a new replica to an alternate node immediately, and even start a new machine if it is required. This enables us to avoid the need to keep many replicas per node, which reduces the total amount of storage and therefore cost associated with it.

You can also control whether the replication should be synchronous or asynchronous, or a combination of the two. This determines the level of consistency, reliability and performance of your cluster. With synchronous replication, you get guaranteed consistency and availability at the cost of performance (a write operation followed by a read operation is guaranteed to return the same version of the data, even in the case of a failure). The most common configuration with GigaSpaces, is synchronous replication to the backup, and asynchronous to the backend storage.

Dynamic Scaling

In order to handle the continuous growth of data, most NOSQL alternatives provide a way of growing your data cluster, without bringing the cluster down or forcing a complete re-partitioning. One of the known algorithms that is used to deal with this, is called consistent hashing. There are various algorithms implementing consistent hashing. 

One algorithm notifies the neighbors of a certain partition, that a node joined or failed. Only those neighbor nodes are impacted by that change, not the entire cluster. There is a protocol to handle the transitioning period while the re-distribution of the data between the existing cluster and the new node takes place. 

Another (and significantly simpler) algorithm uses logical partitions. With logical partitions, the number of partitions is fixed, but the distribution of partitions between machines is dynamic. So for example, if you start with two machines and 1000 logical partitions, you have 500 logical partitions per machine. When you add a third machine, you have 333 partitions per machine. Since logical partitions are lightweight (they are basically a hash table in-memory), it is fairly easy to distribute them. 

The advantage of the second approach is that it is fairly predictable and consistent, whereas with the consistent hashing approach, the distribution between partitions may not be even, and the transition period when a new node joins the network can take longer. A user may also get an exception if the data that he is looking for is under transition. The downside of the logical partitions approach, is that the scalability is limited to the number of logical partitions.

For more details in that regard, I recommend reading Ricky Ho's  post entitled NOSQL Patterns.

Query Support

This is an area where there is a fairly substantial difference between the various implementations.The common denominator is a key/value matching, as in a hash table. Some implementations provide more advanced query support, such as the document-oriented approach, where data is stored as blobs, with an associated list of key/value attributes. In this model you get a schema-less storage that makes it easy to add/remove attributes from your documents, without going through schema evolution etc. With GigaSpaces we support a large portion of SQL. If the SQL query doesn’t point to a specific key, the query is mapped to a parallel query to all nodes, and aggregated at the client side. All this happens behind the scene and doesn’t involve user code.

Use Map/Reduce to Handle Aggregation

Map/Reduce is a model that is often used to perform complex analytics, that are often associated with Hadoop. Having said that, it is important to note that map/reduce is often referred to as a pattern for parallel aggregated queries. Most of the NOSQL alternatives do not provide built-in support for map/reduce, and require an external framework to handle these kind of queries. With GigaSpaces, we support map/reduce implicitly as part of our SQL query support, as well as explicitly through an API that is called executors. With this model, you can send the code to where the data is, and execute the complex query directly on that node.

For more details in that regard, I recommend reading Ricky Ho's post entitled Query Processing for NOSQL DB.

Disk-Based vs. In-Memory Implementation

NOSQL alternatives are available as a file-based approach, or as an in-memory-based approach. Some provide a hybrid model that combines memory and disk for overflow. The main difference between the two approaches comes down mostly to cost/GB of data and read/write performance.

An analysis done recently by Stanford University, called “The Case for RAMCloud” provides an interesting comparison between the disk and memory-based approaches, in terms of cost performance. In general, it shows that cost is also a function of performance. For low performance, the cost of the disk is significantly lower the RAM-based approach, and with higher performance requirements, the RAM becomes significantly cheaper.

The most obvious drawbacks of RAMClouds are high cost per bit and high energy usage per bit. For both of these metrics RAMCloud storage will be 50-100x worse than a pure disk-based system and 5-10x worse than a storage system based on flash memory (see [1] for sample configurations and metrics). A RAMCloud system will also require more floor space in a datacenter than a system based on disk or flash memory. Thus, if an application needs to store a large amount of data inexpensively and has a relatively low access rate, RAMCloud is not the best solution.
However, RAMClouds become much more attractive for applications with high throughput requirements. When measured in terms of cost per operation or energy per operation, RAMClouds are 100-1000x more efficient than disk-based systems and 5-10x more efficient than systems based on flash memory. Thus for systems with high throughput requirements a RAM-Cloud can provide not just high performance but also energy efficiency. It may also be possible to reduce RAMCloud energy usage by taking advantage of the low-power mode offered by DRAM chips, particularly during periods of low activity.  In addition to these disadvantages, some of RAM-Cloud's advantages will be lost for applications that require data replication across datacenters. In such environments the latency of updates will be dominated by speed-of-light delays between datacenters, so RAM-Clouds will have little or no latency advantage. In addition, cross-datacenter replication makes it harder for RAMClouds to achieve stronger consistency as described in Section 4.5. However, RAMClouds can still offer exceptionally low latency for reads even with cross-datacenter replication.

Is it Just Hype?

One of the most common questions that I get this days is: “Is all this NOSQL just hype?”, or “Is it going to replace current databases?"

My answer to these questions is that the NOSQL alternatives didn’t really start today. Many of the known NOSQL alternatives have existed for more than a decade, with lots of successful references and deployments. I believe that there are several reasons why this model has become more popular today. This first is related to the fact that what used to be a niche problem that only a few fairly high-end organizations faced, became much more common with the introduction of social networking and cloud computing. Secondly, there was the realization that many of the current approaches could not scale to meet demand. Furthermore, cost pressure also forced many organizations to look at more cost-effective alternatives, and with that came research that showed that distributed storage based on commodity hardware can be even more reliable then many of the existing high end databases. (You can read more on that here.) All of this led to a demand for a cost effective “scale-first database”. I quote James Hamilton, Vice President and Distinguished Engineer on the AWS team, from one of his articles One Size Does Not Fit All:  

“Scale-first applications are those that absolutely must scale without bound and being able to do this without restriction is much more important than more features. These applications are exemplified by very high scale web sites such as Facebook, MySpace, Gmail, Yahoo, and Amazon.com. Some of these sites actually do make use of relational databases but many do not. The common theme across all of these services is that scale is more important than features and none of them could possibly run on a single RDBMS”

So to sum up – I think that what we are seeing is more of a realization that existing SQL database alternatives are probably not going away any time soon, but at the same time they can’t solve all the problems of the world. Interestingly enough the term NOSQL has now been changed to Not Only SQL, to represent that line of thought.

References

• Recorded Dzone Webinar on NOSQL Alternatives
• NoSQL Alternatives: Dzone Inteview   by Nitin Bharti
• No to SQL? Anti-database movement gains steam – My Take
• NoSQL And Elastic Caching Platforms Are Kissing Cousins by Mike Gaultieri Forrester
• Why Existing Databases (RAC) are So Breakable!
• NoSQL Ecosystem By Jonathan Ellis,
• NoSQL: Distributed and Scalable Non-Relational Database Systems by Jeremy Zawodny
• NOSQL Patterns by Ricky Ho
• Query Processing for NOSQL DB by Ricky Ho
• One Size Does Not Fit All  by James Hamilton
• NOSQL debrief by Johan Oskarsson,
• The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM by Department of Computer Science Stanford University
• GigaSpaces Map/Reduce API

Why Existing Databases (RAC) are So Breakable!

One of the core assumption behind many of today’s databases is that disks are reliable. In other words, your data is “safe” if it is stored on a disk, and indeed most database solutions rely heavily on that assumption. Is it a valid assumption?

Disks are not reliable!

Recent analysis of disk failure patterns (see references below) break the disk reliability assumption completely. Interestingly enough, even though these studies were published in 2007, most people I speak to aren’t aware of them at all. In fact, I still get comments from people who are basing their reliability solution on disk-based persistency, insisting on the argument that whether or not you are reliable amounts to whether or not your data is synched to a disk.

Below are some of the points I gathered from the research, which should surprise most of us:

• Actual disk failure/year is 3% (vs. estimates of 0.5 - 0.9%) – this is a 600% difference on reported vs. actual disk failure. The main reason is that manufacturers can’t tests their disks for years, so their tests are an extrapolation of statistical behavior which proves to be completely inaccurate.
• There is NO correlation between failure rate and disk type – whether it is SCSI, SATA, or fiber channel. Most data centers are based on the assumption that investing in high-end disks and storage devices will increase their reliability – well, it turns out that high-end disks exhibit more or less the same failure patterns as regular disks! John Mitchell had an interesting comment on this matter during our Qcon talk, when someone pointed to RAID disks as their solution for reliability. John said that since RAID is based on an exact H/W replica that lives in the same box, there is a very high likelihood that if a particular disk fails, its replica will fail in the same way. This is because they all have the exact same model, handle the exact same load and share the same power/temperature. 
• There is NO correlation between high disk temperature and failure rates – I must admit that this was a big surprise to me, but it turned out that there is no correlation between disk temperature and disk failure. In fact, most failures happen when disks were at low or average temperature.

Why are existing database clustering solutions so breakable?

Most existing database clustering solutions rely on a shared disk storage to maintain their cluster state, as can be seen in the diagram below.

 

Typical database clustering architecture based on shared storage

The core, implicit assumption behind Oracle RAC, IBM DB2, and many other database clustering solutions, is that failure can be avoided by purchasing high-end disk storage and using expensive hardware (fiber optics, etc). As can be seen from the research I mentioned earlier, this core assumption doesn’t correlate with the failure statistics. Hence I argue that the database clustering model is inherently breakable.

Failure is inevitable – cope with it

Jason McHugh, a senior engineer at Amazon who works on S3, gave a very interesting talk at the Qcon conference (Amazon S3: Architecting For Resieliency In The Face Of Failures), where he outlined the service failure handling architecture of Amazon’s Simple Storage. Jason’s main point was that rather than trying to prevent failure from happening, you should assume that failure is inevitable and instead, design your system to cope with it.

Failure can happen in various shapes and forms:

• Expect drives to fail
• Expect network connections to fail (independent of the redundancy in networking)
• Expect a single machine to go out

Human error example: Failure to use caution while pushing a rack of servers

How to design your application for resilience in the face of failure

Jason outlined some of the lessons from his experience at Amazon:

• Expect and tolerate failures – build your infrastructure to cope with catastrophic failure scenarios: "Distributed systems will experience failures, and we should design systems which tolerate rather than avoid failures. It ends up being less important to engineer components for low failure rates (MTBF) than it is to engineer components with rapid recovery rates (MTTR)." (from Architecture Summit 2008 - Write up)
• Code for large-scale failures – design your application to cope with the fact that services can come and go.
• Expect and handle data and message corruption – don’t rely only on TCP to ensure the consistency of your data. Add validation as part of your application code to make sure that messages are not corrupted.
• Code for elasticity – load/demand may grow unexpectedly. Elasticity is essential to dealing with unexpected demand without bringing the system to its knees. It is also important that when you exceed your capacity, the system won’t crash but instead report denial of service.
• Monitor, extrapolate and react – monitoring at the right level enables you to detect failures before they happen and take the appropriate corrective actions.
• Code for frequent single machine failures – rack failure can cause an immediate disappearance of multiple machines at the same time.
• Game days – simulate failures and learn how to continuously improve your operational procedure for handling failures and architecture.

Memory can be more reliable then disk

Many people assumes that memory is an unreliable data storage.  That assumption holds true if your data “lives” on a single machine; in this case if the machine fails or crashes your application crashes. But what if you distribute the data across a cluster of nodes and maintain more than one copy of the data over the network? In this case, if a node crashes the data is not gone; it lives elsewhere and can be continuously served from one of its replicas.

Under these conditions, I can argue that under certain conditions, memory can be more reliable than disk. One of the reason is the hardware itself – unlike disk drives that rely on mechanical moving parts, RAM is just a silicon chip. As such the chances for failure can be significantly smaller. Unfortunately, I don’t have the data to back up this argument, but I think it’s very likely that this is the case. In an article published on InfoQ titled RAM is the new disk, Tim Bray brings an interesting observation on this matter:

Memory is the new disk! With disk speeds growing very slowly and memory chip capacities growing exponentially, in-memory software architectures offer the prospect of orders-of-magnitude improvements in the performance of all kinds of data-intensive applications. Small (1U, 2U) rack-mounted servers with a terabyte or more or memory will be available soon, and will change how we think about the balance between memory and disk in server architectures. Disk will become the new tape, and will be used in the same way, as a sequential storage medium (streaming from disk is reasonably fast) rather than as a random-access medium (very slow). Tons of opportunities there to develop new products that can offer 10x-100x performance improvements over the existing ones.

As a matter of fact we (GigaSpaces/Cisco) are working these very days on benchmarking  Cisco UCS, where we are evaluating the possibility to manage x100GB and potentially even Terabytes of data in a single node. The results look extremely promising, so the future is not that far ahead.

References

• KHB: Real-world disk failure rates: surprises, surprises, and more surprises by Valerie Aurora (Henson)
• Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? By Bianca Schroeder and Garth A. Gibson
• The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM By Department of Computer Science Stanford University
  
• Amazon S3: Architecting For Resieliency In The Face Of Failures By Jason McHugh
• Failure Trends in a Large Disk Drive Population - Google, Inc., February 2007
• InfoQ Article - RAM is the new disk...
• Are Cloud Based Memory Architectures the Next Big Thing?, highscalability.com