Nati Shalom's Blog

Why Existing Databases (RAC) are So Breakable!

One of the core assumption behind many of today’s databases is that disks are reliable. In other words, your data is “safe” if it is stored on a disk, and indeed most database solutions rely heavily on that assumption. Is it a valid assumption?

Disks are not reliable!

Recent analysis of disk failure patterns (see references below) break the disk reliability assumption completely. Interestingly enough, even though these studies were published in 2007, most people I speak to aren’t aware of them at all. In fact, I still get comments from people who are basing their reliability solution on disk-based persistency, insisting on the argument that whether or not you are reliable amounts to whether or not your data is synched to a disk.

Below are some of the points I gathered from the research, which should surprise most of us:

• Actual disk failure/year is 3% (vs. estimates of 0.5 - 0.9%) – this is a 600% difference on reported vs. actual disk failure. The main reason is that manufacturers can’t tests their disks for years, so their tests are an extrapolation of statistical behavior which proves to be completely inaccurate.
• There is NO correlation between failure rate and disk type – whether it is SCSI, SATA, or fiber channel. Most data centers are based on the assumption that investing in high-end disks and storage devices will increase their reliability – well, it turns out that high-end disks exhibit more or less the same failure patterns as regular disks! John Mitchell had an interesting comment on this matter during our Qcon talk, when someone pointed to RAID disks as their solution for reliability. John said that since RAID is based on an exact H/W replica that lives in the same box, there is a very high likelihood that if a particular disk fails, its replica will fail in the same way. This is because they all have the exact same model, handle the exact same load and share the same power/temperature. 
• There is NO correlation between high disk temperature and failure rates – I must admit that this was a big surprise to me, but it turned out that there is no correlation between disk temperature and disk failure. In fact, most failures happen when disks were at low or average temperature.

Why are existing database clustering solutions so breakable?

Most existing database clustering solutions rely on a shared disk storage to maintain their cluster state, as can be seen in the diagram below.

 

Typical database clustering architecture based on shared storage

The core, implicit assumption behind Oracle RAC, IBM DB2, and many other database clustering solutions, is that failure can be avoided by purchasing high-end disk storage and using expensive hardware (fiber optics, etc). As can be seen from the research I mentioned earlier, this core assumption doesn’t correlate with the failure statistics. Hence I argue that the database clustering model is inherently breakable.

Failure is inevitable – cope with it

Jason McHugh, a senior engineer at Amazon who works on S3, gave a very interesting talk at the Qcon conference (Amazon S3: Architecting For Resieliency In The Face Of Failures), where he outlined the service failure handling architecture of Amazon’s Simple Storage. Jason’s main point was that rather than trying to prevent failure from happening, you should assume that failure is inevitable and instead, design your system to cope with it.

Failure can happen in various shapes and forms:

• Expect drives to fail
• Expect network connections to fail (independent of the redundancy in networking)
• Expect a single machine to go out

Human error example: Failure to use caution while pushing a rack of servers

How to design your application for resilience in the face of failure

Jason outlined some of the lessons from his experience at Amazon:

• Expect and tolerate failures – build your infrastructure to cope with catastrophic failure scenarios: "Distributed systems will experience failures, and we should design systems which tolerate rather than avoid failures. It ends up being less important to engineer components for low failure rates (MTBF) than it is to engineer components with rapid recovery rates (MTTR)." (from Architecture Summit 2008 - Write up)
• Code for large-scale failures – design your application to cope with the fact that services can come and go.
• Expect and handle data and message corruption – don’t rely only on TCP to ensure the consistency of your data. Add validation as part of your application code to make sure that messages are not corrupted.
• Code for elasticity – load/demand may grow unexpectedly. Elasticity is essential to dealing with unexpected demand without bringing the system to its knees. It is also important that when you exceed your capacity, the system won’t crash but instead report denial of service.
• Monitor, extrapolate and react – monitoring at the right level enables you to detect failures before they happen and take the appropriate corrective actions.
• Code for frequent single machine failures – rack failure can cause an immediate disappearance of multiple machines at the same time.
• Game days – simulate failures and learn how to continuously improve your operational procedure for handling failures and architecture.

Memory can be more reliable then disk

Many people assumes that memory is an unreliable data storage.  That assumption holds true if your data “lives” on a single machine; in this case if the machine fails or crashes your application crashes. But what if you distribute the data across a cluster of nodes and maintain more than one copy of the data over the network? In this case, if a node crashes the data is not gone; it lives elsewhere and can be continuously served from one of its replicas.

Under these conditions, I can argue that under certain conditions, memory can be more reliable than disk. One of the reason is the hardware itself – unlike disk drives that rely on mechanical moving parts, RAM is just a silicon chip. As such the chances for failure can be significantly smaller. Unfortunately, I don’t have the data to back up this argument, but I think it’s very likely that this is the case. In an article published on InfoQ titled RAM is the new disk, Tim Bray brings an interesting observation on this matter:

Memory is the new disk! With disk speeds growing very slowly and memory chip capacities growing exponentially, in-memory software architectures offer the prospect of orders-of-magnitude improvements in the performance of all kinds of data-intensive applications. Small (1U, 2U) rack-mounted servers with a terabyte or more or memory will be available soon, and will change how we think about the balance between memory and disk in server architectures. Disk will become the new tape, and will be used in the same way, as a sequential storage medium (streaming from disk is reasonably fast) rather than as a random-access medium (very slow). Tons of opportunities there to develop new products that can offer 10x-100x performance improvements over the existing ones.

As a matter of fact we (GigaSpaces/Cisco) are working these very days on benchmarking  Cisco UCS, where we are evaluating the possibility to manage x100GB and potentially even Terabytes of data in a single node. The results look extremely promising, so the future is not that far ahead.

References

• KHB: Real-world disk failure rates: surprises, surprises, and more surprises by Valerie Aurora (Henson)
• Disk failures in the real world: What does an MTTF of 1,000,000 hours mean to you? By Bianca Schroeder and Garth A. Gibson
• The Case for RAMClouds: Scalable High-Performance Storage Entirely in DRAM By Department of Computer Science Stanford University
  
• Amazon S3: Architecting For Resieliency In The Face Of Failures By Jason McHugh
• Failure Trends in a Large Disk Drive Population - Google, Inc., February 2007
• InfoQ Article - RAM is the new disk...
• Are Cloud Based Memory Architectures the Next Big Thing?, highscalability.com

It's Complex to be Simple

Avdi Grimm wrote an interesting post Simplicity is Complicated. Avdi bring few interesting points on how the term Simplicity is often abused to justify all sort of decisions.

“Everything should be made as simple as possible, but not simpler” - Einstein 

The Meaning of Simplicity

“Simplicity” means different things to different people depending on the context of the problem you’re trying to solve. Avdi provides a useful typology of cases in which the term “simplicity” is often used (and abused): minimizing unnecessary effort, hiding complexity, avoiding difficult-to-understand features, avoiding formal architecture, elegance of design, staying close to the domain.

I’ll spare you the details but I would highly recommend reading the full post. One of the points that I tend to see quite often in various discussions these days is the use of simplicity to justify new language choice by measuring the number of lines of code, elegance as a complexity measurement:

“sometimes when you choose to do less you force someone else to do more. And sometimes that someone else is you, two weeks down the road”

In other words, reducing the number of lines of code (elegance of code) is only one measure of complexity. Other measures include existing skillset, learning curve, available toolset, maturity, interoperability with legacy and other third party tools, etc. Choosing a certain language because it will reduce the amount of code may seem to be logical for a specific area of our application, but at the same time may not be the right choice if you take a broader perspective and look at the entire application or project. At that level, introducing a new language can actually end up introducing huge complexity.

Scalability & Simplicity

Scalability is one of the areas in which I could see almost all sort of the simplicity arguments being abused to justify a certain architectural choice. I have often used Twitter as an illustration for this type of thinking. For example, one of the more critical decisions in any product is the selection of the application framework. Twitter was well known for its selection of Ruby. Simplicity (and productivity which is essentially a synonym of simplicity) was used as the main argument for selecting Ruby vs. Java or any other framework at the time. Twitter made a very similar simplicity argument to justify the selection of Java and Scala over Ruby a year later. It would seem that the term “simplicity” was used to justify a certain choice, and then the exact argument was made to justify almost the complete opposite choice. This is a great illustration of how the simplicity argument becomes a tool to justify almost any choice.

I’ve argued that “simplicity” was an excuse to bypass some basic methodical thinking that could have saved Twitter the painful trial-and-error process. For example, it was a well known fact that Ruby didn’t provide a good concurrency story. In addition, Ruby was lacking many of the performance tuning measurement tools that exist in Java. It shouldn’t be that hard to realize that if your looking for scalability, relying completely on Ruby is going to put some strong limits into your architecture. If you add this to the mix, the question arises: is Ruby really that “simple”? Avdi brings another interesting observation on that regard:

Ruby on Rails made web development simple, relative to tools that came before it. It accomplished this feat by embracing "convention over configuration". In order to do this, it had to incorporate a great deal of extra complexity in the form of algorithms which guess the intent of the programmer instead of forcing her to specify her wishes explicitly. Anyone using it eventually has to become familiar with these hidden rules in order to understand why Rails behaves the way it does.

In other words it is “simple” if your application falls directly under the realm of assumptions of those who designed that framework and could turn out to be complex if you fall outside the boundaries of that realm. So the answer is that you shouldn’t take it for granted that ROR is “simple”.

Another example I’d like to use in this context is GigaSpaces. With GigaSpaces we worked really hard to make scalability simple. Our mission was to enable application to run on a distributed cluster of machines as if it was a single machine. We achieved that by removing the number of moving parts (SBA) and by providing an end-to-end architecture that will solve the scalability of the entire application from the load-balancer to the database. Yet the consistent feedback I heard, at the time we first introduced this concept, was that GigaSpaces is complex. When I asked users for an explanation the answer was quite interesting. Most users defined their problem domain in terms of a database bottleneck. They didn’t define their problem in terms of end-to-end scalability, and they often weren’t even aware that solving only the database bottleneck is not going to make their entire application scalable.

When cloud computing emerged, it forced users to think about elasticity. Elasticity means that you expect that your entire application will scale up or down when a new machine joins or leave the network. Elasticity forced users to look at how they can scale their entire application. All of a sudden, GigaSpaces appeared to be a significantly simpler solution then most of the available alternatives, which relied on static application server clusters and tier-based implementations that didn’t quite fit the dynamic cloud model.

That brings us to another question: What makes the same product “complex” at one point in time and “simple” at another point of time?

Avdi provides an interesting answer to that question:

Complexity is like an air pocket trapped under a layer of airtight plastic. If you push it down it just pops up somewhere else. Most of our attempts to "simplify" software really amount to pushing the complexity somewhere (or some-when) else. And our disagreements about what is meant by simplicity are really about where we think the complexity belongs. ..

In other words it all depends on where you define your area of complexity when you’re making your technology selections. Changing the definition of complexity from “solving database bottleneck”, to “application elasticity” yields very different results.

Final Words

There are tons of examples I can think of where I could see how the same platform or product could turn out to be “simple” for a certain set of challenges and “complex” for other set of challenges. It’s important to note that even though I used Twitter and Ruby as illustrations, the arguments are not specific to those examples and can be applied to almost any language, framework or product be – be it PHP, Java, Spring or even GigaSpaces.

As Avdi pointed out, the goal of “simplicity” can be used to justify just about any decision and therefore we need to be very cautious when we, or somebody else, bring it up to justify a choice. Making a certain complex problem “simpler” is often not enough, if your users define their complexity in a different area. I’ll close with Avdi’s advice:

I challenge myself and anyone who reads this, next time you use the word "simple", to stop and think about what you mean by it. As an exercise, try to rephrase it in terms of where you are pushing the complexity.