I was reading James Hamilton's coverage on Google Megastore: The Data Engine Behind GAE (a must read!), where he summarizes the main architecture assumption behind Google Megastore.
The thing that caught my eyes was the following line:
Support for consistency unusual for a NoSQL database but driven by (what I believe to be) the correct belief that inconsistent updates make many applications difficult to write (see I Love Eventual Consistency but …)
This was later further elaborated upon in one of James' comments below the posting:
The cap theorem is undeniably true but, like many engineering issues, it requires judgment. Consistency can be maintained over remarkably a large scope of data. The availability that CAP forces to be given up certainly exists but can be made quite small. Overall, I would say that CAP encourage many to give up hope way too quickly and build an eventually consistent system when better choices are available with more work. The paper under discussion here shows that a high-scale store can be fully consistent over substantial volumes of data and they wisely decided to not make the entire store globally consistent. Interpreting and applying CAP is a judgement call and we see one approach here. SimpleDB chose to do both (http://perspectives.mvdirona.com/2010/02/24/ILoveEventualConsistencyBut.aspx) which is along the lines you recommend in your conclusion: “as scalable data sources mature they will increasingly and perhaps ubiquitously allow user-selectable CAP choices in much the same way that SQL stores almost always allow user-selectable transaction consistency”. I think you are right.
Interestingly enough James also points to another interesting part of his previous post on the subject (I love eventual consistency but...) where he describes how Amazon deals with consistency tradeoffs with SimpleDB:
What I’ve learned over the years is that strong consistency, if done well, can scale to very high levels. The trick is implementing it well. The naïve approach to achieve full consistency is to route all updates through a single master server but clearly this won’t scale. Instead divide the update space into a large number of partitions, each with its own master. That scales but there is still a tension between the number of partitions and the cost of maintaining many partitions and avoiding hot spots. The obvious way to avoid hot spots is to use a large number of partitions but this increases partition management overhead. The right answer is to be able to dynamically repartition to maintain a sufficient number of partitions and to be able to adapt to load increases on any single server by further spreading the update load.
I found those comments particularly interesting as many people refer to Google and Amazon as the poster children for justifying eventual consistency.
It is interesting to see that the reality is that even Google and Amazon - which I would consider the extreme cases for big data - realized the limitation behind eventual consistency and came up with models that can deal with scaling without forcing a compromise on consistency as I also noted in one of my recent NoCAP series here, here and here.
