In May of last year I wrote my first write-up on Twitter scalability, titled Twitter as a scalability case study. Back then, Twitter was still struggling with its Ruby implementation and claimed to be 10000 Percent Faster. During the past few weeks I came across the an interview entitled Twitter jilts Ruby for Scala which described some of the new development that led Twitter to switch some of its Ruby backend development to Scala. I had follow-on discussions with few colleagues ever since WRT to Scala and functional languages in general which led me to write this piece.
I must admit that whenever a scalability discussion becomes a language-choice discussion, I get irritated. It's enough to read a comment such as the one below to see that there is something is missing from the puzzle:
"Today, Payne said, most of the hip dev set codes in Ruby or PHP or Python because they're perceived as "agile" languages.. but also because the deverati grew bored with languages like Java and C++"
After I read the following note, things became a bit clearer:
"By mid-2008, one of these Ruby message queues completely crashed and developers needed two and a half hours to shove the dropped Tweets back through the system. When your game is micro-blogging, that's a lifetime. Then, in his spare time, one developer ported the code to Scala. According to Payne, the Scala queue could process the same message backlog in 20 seconds."
Later, I read Bill Venner’s excellent interview, Twitter on Scala, which i found quite insightful. But after reading the following response on the use of the Actor model in Twitter I felt even more confused:
"..we found that actors weren’t necessarily the ideal concurrency model for all parts of that system. Some parts of the concurrency model of that system are still actor based. For example, it uses a memcache library that Robey wrote, which is actor based. But for other parts we’ve just gone back to a traditional Java threading model. The engineer working on that, John Kalucki, just found it was a little bit easier to test, a bit more predictable. The nice thing was, it took minutes to switch code that was actor based over to something thread based."
Now to be clear, I have nothing against Scala or Ruby or any of the dynamic languages – quite the contrary. I believe that productivity, being agile, and enjoying writing the code are important factors when we make our language choice. At the same time we should remember that these are only some of the factors; other factors which should influence our decision are available tooling, existing skill-set, maturity, performance, etc. I also don’t care that much how Twitter makes their technology choices. What gets me worried is comments such as this one:
“the switch should stand as a lesson to cutting-edge coders everywhere”
When this type of statements gets to the press and gets backed with intelligent stories that justifies these choices, it is clear to me that many "naive" individuals will start following the same choices without understanding the full picture and history that led to the decision, thinking that they can base their decision on “proven success”. Looking at the history could evil a fairly different picture.
What I found missing in the entire discussion both here in Twitter case and in many of the other web2.0 scalabilty architecture stories is something that will indicate that there is some consistent methodology behind the architectural choices. For example, most of the limitations that were mentioned about Ruby’s threading model, as well as its memory utilization issues, were well known before Twitter chose Ruby as their core language. Why did it take such a long time to realize that you can’t come up with scalable architecture without addressing those limitations? I could argue the exact same thing about Digg and others.
What i would recommend is to look at Brian Zimmer’s summary on Scalability Worst Practices, in particular the “Golden Hammer” point:
"The Golden Hammer refers to the old adage: if all you have is a hammer, everything looks like a nail. Many developers fall prey to the idea of using only one technology – the cost of this is having to build and maintain an infrastructure in the chosen technology which may be readily available in another technology which is more suited to the specific problem area's functionality or abstractions. Forcing a particular technology to work in ways it was not intended is sometimes counter-productive."
The lesson in our specific Twitter case is that language choice should be used as a means for implementing a given architecture and not the other way around. We may find that implementing various parts of our architecture require different technologies and languages, and that’s perfectly fine, as long it follows that order. In Twitter’s case, choosing Ruby for the front-end and Scala and Java for the heavy stuff sounds reasonable, only that I would expect them to get to this realization much sooner.
Final words
In this blog I pointed my comments specifically at Twitter, however Twitter is only an example. Many Web 2.0 sites developed their scalability architecture in a similar trial-and-error approach, which, as we can learn by their histories, tends to be very painful and costly. And yet, building a scalable Web 2.0 application such as Twitter shouldn’t be rocket science. I’m certain that if you follow the right design principles and learn from other proven scalability patterns, you can avoid a large part of the painful “trial and error” experience before coming up with the right solution. Making the right build vs buy decisions can make a lot of difference. In one of my recent posts, Designing a Scalable Twitter, I tried to suggest a methodology for coming up with scalable Twitter architecture. Like anything in life, I’m sure that there is more then one possible solution and more then one language that could be used to implement our solution. In many cases you will find that even though language and platforms could vary between different implementations, the architecture principles remain pretty much the same.
To sum that up, I would say that scalability is first and foremost an architecture choice; different languages can make the implementation of a certain architecture simpler, but language by itself doesn’t make any application more scalable. Different parts of an application may require different set of languages, and choosing the right langrage should also take into consideration existing skillset, maturity, existing development tools and best practices – and obviously, the application’s performance and scalability characteristics.
