Nati Shalom's Blog

Web Speed Can Push You Off of Google Search Rankings! What Can You Do?

Last month Google added Website speed to its site ranking algorithm: It’s Official: Google Now Counts Site Speed As A Ranking Factor.

According to the article there are two primary ways Google will measure page speed:

1. How a page responds to Googlebot
2. Load time as measured by the Google Toolbar

A slow site can translate into a poor user experience and a lower conversion rate -- which in itself can lose you money. But this new development can actually cause you further harm by lowering your Google search ranking, as was recently reported by this users: Google Rankings Drop! Where to Check Site Response Time?

I noticed some of my sites drop significantly for many of the keywords that I rank for in Google. I rank in the top 10 for several keywords (on several different sites) and my traffic came to a screeching halt today. My rankings are down to the 7th or 8th page or lower!

After doing some digging around, I decided to login into my Google Webmaster tools. Sure enough I found a warning next the sitemaps for all my affected sites.
The warning was: "Some URLs in the Sitemap have a high response time.
This may indicate a problem with your server or with the content of the page."

Understanding why Web Speed Matters

The rationale behind this move by Google is fairly straightforward:

Slow web sites lead to a poor user experience, and therefore should not appear at the top of the search list even if they contain relevant content.

This is not something Google made up -- the effects of slow speed, or latency, are well documented.

Matt McGee quotes interesting statistics on the impact of latency on traffic behavior:

Amazon found that every 100ms of latency cost them 1% in sales. Google found that an extra .5 seconds in search page generation time dropped traffic by 20%. James Hamilton (Amazon) provides more references in his post The Cost of Latency.

How to Improve Web Speed

What makes web-speed improvement difficult is that it involves multiple layers, some of which you don’t really control. It depends on the browser and how fast it processes CSS, JavaScript, etc... On the size of your pages and images, where your site is physically located, and the actual server-side architecture.

There is plenty of information on how to achieve better web speed from your browser to your server. Steve Souders book High Performance Web Sites published 14 rules for performance optimization:

• Make Fewer HTTP Requests
• Use a Content Delivery Network
• Add an Expires Header
• Gzip Components
• Put Stylesheets at the Top
• Put Scripts at the Bottom
• Avoid CSS Expressions
• Make JavaScript and CSS External
• Reduce DNS Lookups
• Minify JavaScript
• Avoid Redirects
• Remove Duplicates Scripts
• Configure ETags
• Make Ajax Cacheable

Web Speed at Large Scale 

Steve's 14 rules apply to any site of any size. However, large-scale sites require additional special treatment. Large-scale web sites depend more heavily on the architecture. Non-scalable architecture can lead to devastating results under load. In this section I will discuss how to control web speed at large scale from an application architecture perspective.

Brief History

In Dec 2007 I summarized the main lessons from Google, Amazon, and LinkedIn large-scale web site architecture in this post. On December 8th, 2008, I wrote a response to Todd Hoff's post on highscalability.com, Latency is Everywhere and it Costs You Sales - How to Crush it - My Take. In those articles I tried to provide architecture guidelines on how to control latency in large-scale environments. Most of those lessons still hold true today. In this post I want to update some of those lessons based on recent experience with social networking and the emergence of NoSQL alternatives.

The Emergence of Read/Write Web

During the past two years, social networking has significantly changed the web experience. Today’s web sites deal with viral traffic behavior, as can be seen in the twitter traffic sample below. In addition, most of the content on these sites is now written by external users rather than by the site owner.

These differences in behavior led to a demand for read/write scaling as opposed to read-mostly scaling to deal with continuous scaling demands. This later led to the emergence of the NoSQL alternatives which started to pick up during the first quarter of 2010.

Last week I had the honor of a visit by Cees de Groot and members of his team from Marktplaats/eBay. Cees designed large scale web application for years, specifically in the eCommerce area. He was recently in charge of a design of an Adword Service (see reference here) and is now working on moving their entire site from a database-centric PHP architecture to scale-out architecture in Java.

Here are some of the main takeaways from our discussions on how to improve web performance in this new age of read/write scaling demand:

• Improve Data Speed and Scaling

Data access is probably the most notable area of contention in many sites. In a large-scale system, data contention means that concurrent user access to the same table or data item is serialized due to locking. This makes one user request dependent on the other, and therefore, as the scaling grows it will have more and more significant impact on latency.

In many e-commerce sites, product inventory, product catalog, and user profiles tend to be typical areas where this sort of contention happens. In online gaming sites this would apply to the user profile and also to the gaming table. The important thing to note is that not all of our data is exposed to this level of contention. Understanding where the contention happens is the first step in solving the problem.

There are various option to reduce the contention points depending on the access pattern:

Read Mostly

In a read-mostly scenario, many users try to fetch the same content at the same time. Only a few users actually update the content, and even fewer share the content and try to update it concurrently. A large portion of large-scale web sites use memcache today to handle their read-mostly scaling scenarios. Memcache is extremely simple and it exposes the key/value store API. At the same time, Memcache is fairly limited as it doesn't provide consistency and high availability, and therefore cannot be used as a system of record. This means that update operations still need to go through the database, making memcache suitable for read-mostly scenarios but not suitable for write scaling scenarios.

Write-Intensive with High Latency

A write-intensive scenario means that the insert/update rate reaches fairly high levels (compared with read-mostly scenarios). It doesn't necessarily mean that the write rate is higher then the read rate, but rather that it's high enough to hit the limit of the database. Many social networking sites fall under this category as most of the site content is driven by users and not by the site owner.

NoSQL alternatives such as Cassandra can manage write scalability, but in most cases at the expense of consistency. With NoSQL alternatives, getting consistency between write/read comes at the expense of read latency (to ensure read consistency you need to read multiple copies of the same data from all the replicas). Furthermore, write latency is still bound to disk. So with NoSQL we can remove the scaling overhead of read/write but we don't come close to the latency that a memory-based solution such as memcache provides. A good reference to the type of performance that you could expect from some of the file-based NoSQL alternatives is provided here and here.

Read/Write-Intensive with Low latency

In this scenario it is not enough to manage the scaling of our write/read operation -- we need to be able to reduce the time it takes to perform the actual read/write operation. A good example is Twitter. With Twitter, read latency provided by NoSQL alternatives could be too slow to meet overall performance goals.

The solution would be very similar to the one provided by file-based NOSQL alternatives, only that it would be entirely based on memory .

The emergence of large memory devices such as Cisco UCS makes it possible to store Terra bytes of data purely in-memory (see Memory is the New Disk). Unlike memcache, an in-memory data grid such as the one provided by GigaSpaces turns memory devices into a transactional data store that can act as the system of record. This makes them suitable for handling both read and write scaling at extremely low latency.

You can read a more detailed description on how an in-memory data grid can be used for read/write scaling of existing databases in my Scaling Out MySQL post.

How Can I Manage Read/Write Scaling if I'm Already Using Memcache?

Write scaling pressure seem to be pushing sites like Twitter, Digg and others toward NoSQL as a replacement for memcache + MySQL as noted in this article: MYSQL AND MEMCACHED: END OF AN ERA? 

Having said that, many of the sites are already heavily invested in memcache so the implication of that transition translates into fairly significant rewrites.

One way to avoid this rewrite exercise would be to turn memcache into a transactional data-store just as its close in-memory data grid relatives.

Because memcache is basically just a client/server protocol, we can easily add memcache support to an existing data-grid. In this way we can use memcache as a system of record that can manage both read & write scaling. As a matter of fact, we're just about to announce our first memcache support for GigaSpaces for this exact purpose. Other data-grid providers are expected to announce their support for memcache as well.

• Improve Dynamic Page Load Time

Web page content is derived from many sources. The use of asynchronous calls to those services makes it possible to parallelize page rendering, reducing the time it takes to build the entire page content significantly. The diagram below illustrates how parallel page part fetching works:

Sequential Page Part Fetching

Parallel Page Part Fetching

 

While all this makes sense, writing the actual code to do this sort of parallel fetching might not be as trivial as it seems. One of the things that can make this work much simpler is the use of Future. The following snippet taken from GigaSpaces documentation illustrates what this API looks like:

 

The Sync mode is where you call an executor to fetch content from a remote service and then use the Future handle to poll for the result of that call at some other point in time. Even though the user would call the execute method sequentially it wouldn’t be blocked for its execution and therefore the actual execution will happen in parallel.

In the A-Sync mode we are not polling for the result but instead we will get a call-back with the actual results. This could be an ideal way to combine async execution with Ajax, where the callback method could be used to update our page asynchronously after the user already loaded the page given the user a fairly low latency experience.

• Use On-Demand Scaling to Ensure Latency Under Load

Viral traffic behavior means that we can’t predict the load. Or to be more precise, we need to be ready to change our site sizing more frequently then we were previously accustomed. The current practice of over-provisioning based on the busiest hour of the busiest day doesn't hold up anymore: 1) You end up with a huge investment to meet the peak load traffic, 2) Provisioning for peak load leads to average-low utilization during regular hours. To be able to handle this type of traffic behavior we need to design our site for on-demand scaling. On-demand scaling involves the following steps:

Monitor the current traffic

If latency grows beyond a certain threshold, add another web container and update the load-balancer with the IP address of that container. You can read more on how you can take an existing Java web application and add dynamic scaling without changing a single line of code here. 

Control your user traffic

Whether or not you managed to add dynamic scaling, you are always going to be bound by the amount of physical resources you currently have. One of the worst thing that could happen is that your site will crash as a result of an unpredictable peak load. If the Google search bot happens to visit your site while it is down, Google will “punish” your site severely and remove it completely from  search results for quite some time (this is one of the SEO experts' worst nightmares, I've witnessed this on several occasions as my wife is becoming an SEO expert herself). It is therefore considered a best practice to control your user traffic and put a limit on how users can access your site. For example, Twitter limits the size of your tweets, and the number of tweets each user can post in an hour. This is also a good practice to protect your site from malicious attacks. In other words, it is better to send denial of service to some of your users then to lose them all.

Final Notes

In brick and mortar stores we've known for a long time that slow customer service will turn customers away. The retail industry has put a lot of effort into improving customer service and reducing the time customers spend waiting in queues or not being answered. Not surprisingly, we're seeing analogous developments with web site traffic. Slow sites turn customers away. Various measures such as the one presented by Google shows a direct correlation between web site latency and user traffic behavior. Lower user traffic translates immediately to fewer purchases on your site and therefore loss of potential revenue.

Google made a right move by adding web-speed to the search engine ranking. As users, this will help ensure that we get better service from sites that want to be at the top of the search list. For site owners, this places site performance as a much higher priority. For us techies, this is also good news. It will make our work of justifying why we should use better architecture easier as we can easily measure how our work translates to real business value.

References

• It’s Official: Google Now Counts Site Speed As A Ranking Factor

• The Cost of Latency by James Hamilton's

• LATENCY IS EVERYWHERE AND IT COSTS YOU SALES - HOW TO CRUSH IT

• Architecture You Always Wondered About: Lessons Learned at Qcon

• Memcached @ Facebook (Video Cast)

• MYSQL AND MEMCACHED: END OF AN ERA?

• Choosing a key-value storage system (Cassandra vs. Voldemort)

• Writing your own real-time search engine and Adwords service

• Scaling your web application

Lessons from Apple vs. Microsoft vs. Google Product Strategy

I was reading an interesting blog post by Thomas Fitzgerald  on the difference between Microsoft and Apple when it comes to new products. This analysis touches on one of the more fundamental debates in product strategy –- going after market share vs. going after profit. In this post, I've tried to lay out my main take-away from this and other related discussions about the different product strategies, specifically in the areas of market share vs. profit, commoditization, and the role of open source.

 

Innovate vs. Commoditize

Apple and Microsoft provide good examples of these two different strategies, as Thomas Fitzgerald commented in his post:

Apple identifies technologies that are in their infancy or are relatively unexploited and finds away to give them mass market appeal. Examples: iPod, iPhone, The GUI, iTunes, Multitouch

Microsoft on the other hand identifies technologies that are established and successful, then decides that it want’s part (or all) of that success and uses its size and power to chip away at the competition until it gains market share. Examples: Xbox, Bing, MSN, Windows, Windows Mobile, Zune.

The reason I am paying a premium for Apple’s products is because of the wonderful integration of the softwares and hardwares, controlled and maintained by the same company. Everything works like a charm and should a problem arises, it is taken care of directly by Apple. By the way, after more than two decades of using Macs, I have had very few incidents when I needed to contact them

To sum this up in my own words: Innovation involves huge initial investment and risk. On the other hand, successful innovation can open an untapped market which puts you ahead of the competition.

With a commoditization strategy, you let others take the risk of finding a new market, and wait until that market reaches a certain size and maturity. Then, you can penetrate the market with a communization strategy that basically gives the user the same set of product features at significantly lower cost, or even for free. You can afford to, because you didn't carry the cost and the risk involved in the early stages of that market, and you count on volume to compensate for lower profit margins.

Profit vs. Market Share

Apple almost always seems to approach new products with the goal of making a profit from whatever that product may be, whereas Microsoft seems to approach a product line with the goal of dominating that market regardless of profitability.

If you're going after market share, then profit becomes secondary to your strategy. Your first goal is to reach a certain adoption level, and then you can worry about ways to monetize the adoption. If you're going after profit, you’ll first look for what brings the most value to your customers.

The Open Source Model as a Commoditization Strategy

Because of its size and existing market share, Microsoft's commoditization strategy can only apply to Microsoft. New and small companies cannot rely on this model. This is why the open source model emerged.

A good example for this discussion is Oracle vs. MySQL –- or BEA vs. JBoss. Both Oracle and BEA were pioneers in their domain and invested years in creation of the RDBMS and App Server markets. Small companies like MySQL and JBoss needed something more disruptive to get sufficient adoption within existing markets. This is when the open source model started to take off. At the heart of it, what open source gave MySQL and JBoss is basically a “cheap marketing channel". It enabled relatively small companies with low budgets to get significant exposure without the investment usually associated with non-open source products.

Commoditization is Not Risk-Free

Earlier in this post I commented that with a commoditization strategy, you do not share the market innovation and penetration risk and costs. As a result, you gain an advantage over existing players, enabling you to offer a lower-cost product and disrupt the market. For years, this strategy worked pretty well. However, in recent years we’ve witnessed how Microsoft has been threatened by its own commoditization weapon from companies like Red Hat, who offered a “free” operating system and competed head to head with Microsoft's core product –- Windows. 

Red Hat is now also being threaten by other open source alternatives such as CentOS, Ubuntu, and MySQL, as well as by other companies such as VMware who bought SpringSource (a direct competitor of JBoss).

There is something fairly similar between Microsoft and Redhat in this regard. Both used a commoditization strategy to get their share of the market. Both didn’t invest in any innovation for years: instead, they specialized in commoditizing and monetizing other companies' (and individuals' in the case of Red Hat/JBoss) innovative products.

Both Microsoft and Red Hat should serve as a reminder that commoditization is not really a risk-free business. Both Microsoft and Red Hat are now paying for years of lack of investment in innovation, and they are being forced to make much bigger investments now just to catch up with the market and defend their right to exist, costing them more than it would likely have cost them if they had continued investing in innovation.

Innovation Strategy: Attack Is the Best Defense

Most people think of innovation as technology innovation i.e. coming up with new patent or technology breakthrough. Its important to note that innovation can come in different shapes and forms. A business innovation such as the one introduced SalesForce/SaaS model could equally if not more important.

One of the unique things about Apple is that it figured out ways to couple together technology innovation with business innovation (Touch Screen with ITunes and AppleStore). It is classic case where the whole is greater than the sum of its parts.

The tricky thing about innovation is that it involves forward looking i.e. seeing what the customer needs before he even knows it.

Shaun Smith articulate that very well in his article Why Steve Jobs doesn’t listen to customers. Quoting from the article:

..seeing under the surface of what customers want now; they just don’t realize it until they see it.  ..People like Jobs can see what the market is ready for before the market knows it itself.

In most cases asking your customer what they want wouldn’t lead you to the next cycle of innovation:

..customers who provide most of your profit today are those whose views you pay most attention to. Yet ask them what they want and it is likely to be the usual thing – better, faster, cheaper.  They want a better version of what they’ve already got.

Henry Ford supposedly said if he had asked his customers what they wanted, before coming up with the Model T, they’d have asked for a faster horse… Tom Peters and others have long pointed out that the customers least likely to help you move on to the next innovation are the biggest customers you have.

Apple, and to a lesser degree Google, have shown that innovation is an effective tool to beat competition and disrupt the market, more than just giving things for free. I would even argue that Apple's way of innovating has contributed more to the community than if Apple had simply provided free products without a clear use case behind them (such as Google Wave). Which brings me to the difference between Apple and Google...

The Google Model -- Combining Innovation and Commoditization

Google used a different plan of attack, combining commoditization and innovation strategies. Unlike many other companies, Google could afford to penetrate new markets without worrying about how to turn it into a revenue generating business, as Microsoft or Apple would have to. They rely on their existing advertising revenue channel to fund their new business and distrupt the market with an Internet-based (royalty-free) product offering. They rely on the content being generated by those services to generate indirect revenue and rely on the community to help them define the product.

At first sight the Google model seem to be a classic win-win model. You get services for free while Google gets their share as well. However, there is a small catch here: The Google model applies only to Google, and therefore represents a sort of “unfair competition”. The seeming advantage can quickly become a threat to that same community. What motive would Google have to innovate and provide high-quality products if it becomes a monopoly, working in a market with no competition…  There have been several instances recently (Buzz, Wave) that can be very alarming. Michael Gray's post, Is Google Stealing Your Content and Hijacking Your Traffic, taught me that we are probably already more vulnerable already than we think.

Innovation Is Not Enough

Both Apple and Google can be considered as fairly innovative companies. However, there is still a fundamental difference between Apple's way of doing innovation and Google's. The difference lies in the fact that Apple innovates in things are that are core to its business, and which generate direct revenue to justify the investment. While Google innovates on things that indirectly contribute to its business.

The result is that when Apple releases a product it comes out polished and well-scoped, because Apple knows that at the price they charge people will have low tolerance for defects and a clumsy user experience. Compare this to Google Buzz, Wave, Docs, AppEngine, and other Google products, where this is clearly not the case. Google releases half baked products and lets the community debug and develop them.

Sun is another good example of an extremely innovative company with tons of new open source community projects and technologies. Sun failed to turn all this innovation into profit, and at some point when the SPARC “cash cow” stopped printing money, Sun couldn’t cover all those  projects and was forced to shut many of them down. We all know where that ended...

By examining the example of Apple vs. Google, it's easy to see that from a product strategy standpoint innovation for the sake of innovation is not enough. Innovation that doesn’t contribute directly to your bottom line is probably going to be abandoned at some stage, or just go through a long maturity cycle that might do your business more harm than good.

The Jury Is Still Out

The software industry is still fairly young and tends to be very trendy. We tend to jump from one model to another fairly quickly, often without asking very many questions or even trying to learn from past experience, just because it's cool or because someone else that we all know did it…

Apple, Microsoft, and Google represent fairly different product strategy models with varying advantages and limitations that we can learn from. David Cook provided a good summary of these three models in his post Google vs. Microsoft vs. Apple: 3 Development Models for Google Wave:

• Microsoft:  “Everybody gets a say and we listen to everybody and then…we build a product that will do everything.”
• Apple: “They tell you what you need, they build it their way, it does what they say it can do and not do, they make it cool and make sure it works with all their other stuff.”
• Google: “They took a little bit of the Apple model. They went far, far away to a secret island and designed the product without letting anyone find out about it and had a big huge launch. Then they took the MS approach and asked everybody to think of what they wanted improved and what new functionality they would like to see”

The open source model, such as that used by MySQL and JBoss, is another model that applies mostly to smaller startups is another good example of a product strategy. 

My main take-aways from these examples are:

• Innovation that brings immediate value to your customers and business should be the heart of any product strategy.
• Open source can be an effective commoditization and marketing tool, especially if you are a small company but cannot define your long term strategy.
• I would be cautious of using any product (innovative as it may be) that doesn’t have a clear impact on the business bottom-line of the owner, as the chances that it will be abandoned at some point is fairly high.

The jury is still out on the question of product strategy. There are fundamentally different models that are successful, such as Microsoft, Google, Apple, and Oracle.... with no clear winner.

VMware, with its recent set of acquisitions, seem to to be following Google's path, using its virtualization stronghold to fund their new cloud strategy. Where this strategy will lead is still an open question: Will it be an attempt to create a new monopoly or an open ecosystem?

I'll close with part of David Cook's summation:

Perhaps incorporating the business model of MS, Apple and Google is the right road to be on…. Doing any part of that trinity half-hearted or with less enthusiasm will be to the detriment of the whole product..

Microsoft and Red Hat taught us the lesson that a commoditization strategy without innovation is not risk-free, while at the same time innovation just for the sake of innovation -- as with the case of Sun and to a lesser degree Google -- can be just as harmful.

References

• Enterprise 2.0 Roll-up: The Great Google vs. Microsoft Debate
• Google vs Microsoft vs Apple: 3 Development Models for Google Wave
• Is Google Stealing Your Content and Hijacking Your Traffic
• Why Steve Jobs doesn’t listen to customers