Nati Shalom's Blog

It's Complex to be Simple

Avdi Grimm wrote an interesting post Simplicity is Complicated. Avdi bring few interesting points on how the term Simplicity is often abused to justify all sort of decisions.

“Everything should be made as simple as possible, but not simpler” - Einstein 

The Meaning of Simplicity

“Simplicity” means different things to different people depending on the context of the problem you’re trying to solve. Avdi provides a useful typology of cases in which the term “simplicity” is often used (and abused): minimizing unnecessary effort, hiding complexity, avoiding difficult-to-understand features, avoiding formal architecture, elegance of design, staying close to the domain.

I’ll spare you the details but I would highly recommend reading the full post. One of the points that I tend to see quite often in various discussions these days is the use of simplicity to justify new language choice by measuring the number of lines of code, elegance as a complexity measurement:

“sometimes when you choose to do less you force someone else to do more. And sometimes that someone else is you, two weeks down the road”

In other words, reducing the number of lines of code (elegance of code) is only one measure of complexity. Other measures include existing skillset, learning curve, available toolset, maturity, interoperability with legacy and other third party tools, etc. Choosing a certain language because it will reduce the amount of code may seem to be logical for a specific area of our application, but at the same time may not be the right choice if you take a broader perspective and look at the entire application or project. At that level, introducing a new language can actually end up introducing huge complexity.

Scalability & Simplicity

Scalability is one of the areas in which I could see almost all sort of the simplicity arguments being abused to justify a certain architectural choice. I have often used Twitter as an illustration for this type of thinking. For example, one of the more critical decisions in any product is the selection of the application framework. Twitter was well known for its selection of Ruby. Simplicity (and productivity which is essentially a synonym of simplicity) was used as the main argument for selecting Ruby vs. Java or any other framework at the time. Twitter made a very similar simplicity argument to justify the selection of Java and Scala over Ruby a year later. It would seem that the term “simplicity” was used to justify a certain choice, and then the exact argument was made to justify almost the complete opposite choice. This is a great illustration of how the simplicity argument becomes a tool to justify almost any choice.

I’ve argued that “simplicity” was an excuse to bypass some basic methodical thinking that could have saved Twitter the painful trial-and-error process. For example, it was a well known fact that Ruby didn’t provide a good concurrency story. In addition, Ruby was lacking many of the performance tuning measurement tools that exist in Java. It shouldn’t be that hard to realize that if your looking for scalability, relying completely on Ruby is going to put some strong limits into your architecture. If you add this to the mix, the question arises: is Ruby really that “simple”? Avdi brings another interesting observation on that regard:

Ruby on Rails made web development simple, relative to tools that came before it. It accomplished this feat by embracing "convention over configuration". In order to do this, it had to incorporate a great deal of extra complexity in the form of algorithms which guess the intent of the programmer instead of forcing her to specify her wishes explicitly. Anyone using it eventually has to become familiar with these hidden rules in order to understand why Rails behaves the way it does.

In other words it is “simple” if your application falls directly under the realm of assumptions of those who designed that framework and could turn out to be complex if you fall outside the boundaries of that realm. So the answer is that you shouldn’t take it for granted that ROR is “simple”.

Another example I’d like to use in this context is GigaSpaces. With GigaSpaces we worked really hard to make scalability simple. Our mission was to enable application to run on a distributed cluster of machines as if it was a single machine. We achieved that by removing the number of moving parts (SBA) and by providing an end-to-end architecture that will solve the scalability of the entire application from the load-balancer to the database. Yet the consistent feedback I heard, at the time we first introduced this concept, was that GigaSpaces is complex. When I asked users for an explanation the answer was quite interesting. Most users defined their problem domain in terms of a database bottleneck. They didn’t define their problem in terms of end-to-end scalability, and they often weren’t even aware that solving only the database bottleneck is not going to make their entire application scalable.

When cloud computing emerged, it forced users to think about elasticity. Elasticity means that you expect that your entire application will scale up or down when a new machine joins or leave the network. Elasticity forced users to look at how they can scale their entire application. All of a sudden, GigaSpaces appeared to be a significantly simpler solution then most of the available alternatives, which relied on static application server clusters and tier-based implementations that didn’t quite fit the dynamic cloud model.

That brings us to another question: What makes the same product “complex” at one point in time and “simple” at another point of time?

Avdi provides an interesting answer to that question:

Complexity is like an air pocket trapped under a layer of airtight plastic. If you push it down it just pops up somewhere else. Most of our attempts to "simplify" software really amount to pushing the complexity somewhere (or some-when) else. And our disagreements about what is meant by simplicity are really about where we think the complexity belongs. ..

In other words it all depends on where you define your area of complexity when you’re making your technology selections. Changing the definition of complexity from “solving database bottleneck”, to “application elasticity” yields very different results.

Final Words

There are tons of examples I can think of where I could see how the same platform or product could turn out to be “simple” for a certain set of challenges and “complex” for other set of challenges. It’s important to note that even though I used Twitter and Ruby as illustrations, the arguments are not specific to those examples and can be applied to almost any language, framework or product be – be it PHP, Java, Spring or even GigaSpaces.

As Avdi pointed out, the goal of “simplicity” can be used to justify just about any decision and therefore we need to be very cautious when we, or somebody else, bring it up to justify a choice. Making a certain complex problem “simpler” is often not enough, if your users define their complexity in a different area. I’ll close with Avdi’s advice:

I challenge myself and anyone who reads this, next time you use the word "simple", to stop and think about what you mean by it. As an exercise, try to rephrase it in terms of where you are pushing the complexity.

Twitter as a scalability case study – it’s the architecture, stupid!

In May of last year I wrote my first write-up on Twitter scalability, titled Twitter as a scalability case study. Back then, Twitter was still struggling with its Ruby implementation and claimed to be 10000 Percent Faster. During the past few weeks I came across the an interview entitled Twitter jilts Ruby for Scala which described some of the new development that led Twitter to switch some of its Ruby backend development to Scala. I had follow-on discussions with few colleagues ever since WRT to Scala and functional languages in general which led me to write this piece.

I must admit that whenever a scalability discussion becomes a language-choice discussion, I get irritated. It's enough to read a comment such as the one below to see that there is something is missing from the puzzle:

"Today, Payne said, most of the hip dev set codes in Ruby or PHP or Python because they're perceived as "agile" languages.. but also because the deverati grew bored with languages like Java and C++"

After I read the following note, things became a bit clearer:

"By mid-2008, one of these Ruby message queues completely crashed and developers needed two and a half hours to shove the dropped Tweets back through the system. When your game is micro-blogging, that's a lifetime. Then, in his spare time, one developer ported the code to Scala. According to Payne, the Scala queue could process the same message backlog in 20 seconds."

Later, I read Bill Venner’s excellent interview, Twitter on Scala, which i found quite insightful. But after reading the following response on the use of the Actor model in Twitter I felt even more confused:

"..we found that actors weren’t necessarily the ideal concurrency model for all parts of that system. Some parts of the concurrency model of that system are still actor based. For example, it uses a memcache library that Robey wrote, which is actor based. But for other parts we’ve just gone back to a traditional Java threading model. The engineer working on that, John Kalucki, just found it was a little bit easier to test, a bit more predictable. The nice thing was, it took minutes to switch code that was actor based over to something thread based."

Now to be clear, I have nothing against Scala or Ruby or any of the dynamic languages – quite the contrary. I believe that productivity, being agile, and enjoying writing the code are important factors when we make our language choice. At the same time we should remember that these are only some of the factors; other factors which should influence our decision are available tooling, existing skill-set, maturity, performance, etc. I also don’t care that much how Twitter makes their technology choices. What gets me worried is comments such as this one:

“the switch should stand as a lesson to cutting-edge coders everywhere”

When this type of statements gets to the press and gets backed with intelligent stories that justifies these choices, it is clear to me that many "naive" individuals will start following the same choices without understanding the full picture and history that led to the decision, thinking that they can base their decision on “proven success”. Looking at the history could evil a fairly different picture.

What I found missing in the entire discussion both here in Twitter case and in many of the other web2.0 scalabilty architecture stories is something that will indicate that there is some consistent methodology behind the  architectural choices. For example, most of the limitations that were mentioned about Ruby’s threading model, as well as its memory utilization issues, were well known before Twitter chose Ruby as their core language. Why did it take such a long time to realize that you can’t come up with scalable architecture without addressing those limitations? I could argue the exact same thing about Digg and others.

What i would recommend is to look at Brian Zimmer’s summary on Scalability Worst Practices, in particular the “Golden Hammer” point:

"The Golden Hammer refers to the old adage: if all you have is a hammer, everything looks like a nail. Many developers fall prey to the idea of using only one technology – the cost of this is having to build and maintain an infrastructure in the chosen technology which may be readily available in another technology which is more suited to the specific problem area's functionality or abstractions. Forcing a particular technology to work in ways it was not intended is sometimes counter-productive."

The lesson in our specific Twitter case is that language choice should be used as a means for implementing a given architecture and not the other way around. We may find that implementing various parts of our architecture require different technologies and languages, and that’s perfectly fine, as long it follows that order. In Twitter’s case, choosing Ruby for the front-end and Scala and Java for the heavy stuff sounds reasonable, only that I would expect them to get to this realization much sooner.

Final words

In this blog I pointed my comments specifically at Twitter, however Twitter is only an example. Many Web 2.0 sites developed their scalability architecture in a similar trial-and-error approach, which, as we can learn by their histories, tends to be very painful and costly. And yet, building a scalable Web 2.0 application such as Twitter shouldn’t be rocket science. I’m certain that if you follow the right design principles and learn from other proven scalability patterns, you can avoid a large part of the painful “trial and error” experience before coming up with the right solution. Making the right build vs buy decisions can make a lot of difference. In one of my recent posts, Designing a Scalable Twitter, I tried to suggest a methodology for coming up with scalable Twitter architecture. Like anything in life, I’m sure that there is more then one possible solution and more then one language that could be used to implement our solution. In many cases you will find that even though language and platforms could vary between different implementations, the architecture principles remain pretty much the same.

To sum that up, I would say that scalability is first and foremost an architecture choice; different languages can make the implementation of a certain architecture simpler, but language by itself doesn’t make any application more scalable. Different parts of an application may require different set of languages, and choosing the right langrage should also take into consideration existing skillset, maturity, existing development tools and best practices – and obviously, the application’s performance and scalability characteristics.