Nati Shalom's Blog

Cool Projects on OpenSpaces.org

The OpenSpaces.org community site launched in January. I was surprise by the rapid adoption of OpenSpaces since then, with lots of interesting innovations on things I didn't even think of. I'm sure that some of the projects will be very useful to many OpenSpaces users. This shows the value behind  an ecosystem and community. Given the right tools, people will start collaborating and share things that otherwise would be buried in their hard disk, or in their mind.

The OpenSpaces.org site also provides a great tool for GigaSpaces Partners and individuals in the general developer community to expose their skills by publishing valuable content. A good example is GridDynamics, a GigaSpaces partner, who invested time and effort on producing high quality, well-documented projects.

The same goes for various people on the GigaSpaces team who came up with great ideas based on work that they did with customers. They use the OpenSpaces,org platform to share the tools they developed with other users in the community who might have similar needs. For example, the OpenSpaces demos project shows how to integrate Ajax, Spring MVC and OpenSpaces to scale a typical web application (market data front end, in this specific case). 

Another good example is TGris, an extension of the testing grid framework that we use internally at GigaSpaces, and which several customers showed interest in for automating the testing of their own applications (note that the tool is not specific to OpenSpaces).

Another class of  interesting projects are those that integrate OpenSpaces with various frameworks and APIs. These projects simplify the integration and adoption process, and shorten time-to-value. Good examples are the projects that provide integration for OpenSpaces/GigaSpaces with Amazon SimpleDB, JPA, and Memcached , as well as the  Cache Integration project, which enables OpenSpaces/GigaSpaces support for many frameworks, such as Acegi Security, Cocoon, Jetty, iBatis, OpenJPA, Velocity and others.

Other people built entire functional applications,  such as Leonardo Gocalves's  GoDo - Goods Donation System (see details below), and Jim Liddle's MobileGSFeed, which provides a scalable solution for handling Atom feeds through the iPhone. Jim actually runs our Sales in the UK & Ireland. Never in my dreams did I imagine that OpenSpaces.org would be used by sales guys :-)

Anyway, I'm very pleased to let you know that we reached an important milestone for OpenSpaces two weeks ago when we reached the deadline of the developer contest. Fourteen candidates made it to the final stages. Only three will be finalists. A distinguished panel of judges interviewed each contestant. The judges are Adrian Colyer, CTO, SpringSource; Joe Ottinger, Editor, TheServerSide.com; John Davies; Julian Brown, Architecture Consultant, RWE;  Keerat Sharma, Platform Engineer, Gallup; and Ross Mason, Co-founder and CTO, MuleSource.

All of the candidates put up a real good fight and made it very hard for the judges to reach their final decision. The winners of the contest will be announced in a nice venue in Prague during TheServerSide Java Symposium event. Stay tuned for updates on the exact date and venue here and on The GigaSpaces Blog and web site. We also intend to publish interviews with each of the finalist project owners and post them in a blog.

Here are some of the interesting projects (in alphabetical order). The full list of projects can be found here.

• Convergence - integration of Compute Grids (e.g., DataSynapse) with In-Memory Data Grid (IMDG). By Kirill Uvaev
•  DomainProxy - An extension enabling rich domain models with domain relationships to be transparently used with OpenSpaces.  Essentially, a JPA implementation on top of  spaces. By Jason Carreira.
• Examples of Dynamic Scripting with Groovy Several examples of dynamic scripting with  Groovy, which demonstrate various techniques of using Groovy with GigaSpaces XAP. By Ed Crandell
• Amazon SimpleDB as External Data Source - Amazon SimpleDB as an external data source for OpenSpaces persistence. An alternative to Hibernate backed data source. By Chen Wang
• Gigapult -  A JRuby-based domain-specific language designed to specify all aspects of grid configurations in an intuitive and readable way. By  Kirill Ishanov
• GigaSpaces BPEL Engine An engine for executing BPEL (Business Process Execution Language) standard workflows on top of GigaSpaces XAP. By Mihai Lucian
• GoDo - Goods Donation System - A distributed application that connects people who wish to make donations with those in need of the donations. By  Leonardo Goncalves
• Maven 2 OpenSpaces Tools - Maven plug-in allowing GigaSpaces application developers to easily build, assemble and deploy OpenSpaces applications. By Jin Kim
• OpenSpaces Binary Calculator - An application that uses experimental approach for capacity measurement. By Victor Samoylov
• PackRat - A library that will help increase space capacity by packing the Entities. By Alexander Kuznetsov
• SpaceMemcached - Memcached implementation based on Grizzly (NIO framework) and GigaSpaces (as a distributed cache). By Steve Tzou
• TGris - A distributed testing grid that runs on top of GigaSpaces XAP. By Igor Goldenberg
• Web Hits Aggregator - Connector for PHP  - A generic connector for PHP that supports GigaSpaces XAP capabilities, including Take, TakeMultiple, Read, ReadMultiple, Write, Lease and Transaction. By Moshe Kaplan 

Please join one of the projects or start a new one yourself. If you already developed something, but are concerned about the time it will take to initiate a new project -- don't be! It is extremely easy and quick to start a new project and if you need any help, we're ready to support you.

 

 

 

 

Nice article on Space-Based Architecture

I Just came across a very good article by Clara Ko summarizing the concept behind Space-Based Architecture on Java Pulse. The article is based on our recent white paper Scaling Spring Application In 4 Steps.  Clara's article is available here. I highly recommend reading it.

I particularly liked the way Clara summarized the problem with tier-based architectures:

GigaSpaces argues that tier-based architecture is the nemesis to linear scalability. In fact, the architecture itself is the bottleneck, not unoptimized component. This is because in a tier-based architecture, as scability patches are introduced, the communication between the tiers become more complicated and expensive. The complex interaction between tiers and between physical machines in the deployment cause problems with latency and data consistency. Possible problems with tier-based architecture are bottlenecked access to the database, bottlenecked access to a centralized messaging provider, messaging overhead, network latency, inefficient clustering, and unreliable failover recovery. GigaSpaces addresses each of those problems by providing a platform that enables space-based architecture.

Couldn't say it any better.

GigaSpaces Dashboard Using Hyperic plugin

 Alexey Kharlamov created  a project that implements a Hyperic plugin for GigaSpaces. It is available as an open source project on OpenSpaces.org.

Hyperic is open source software for web infrastructure monitoring and management. Among its customers are Hi5 and Contegix. It supports Apache, JBoss, Tomcat, Spring, Weblogic and many other products.

I know of many of GigaSpaces users and customers who expressed interest in having such a customizable monitoring tool.

Now is your chance to join this project, use it, extend it and optimize it.

You can find details of this work here:

http://support.hyperic.com/confluence/display/hypcomm/Gigaspaces

This project also shows the value of OpenSpaces.org as a tool for extending the ecosystem around  the OpenSpaces development framework and GigaSpaces XAP.

 Alexey, very impressive work!

 

Google App Engine - what about existing applications?

Recently, Google announced Google App Engine, another announcement in the rapidly growing world of cloud computing. The announcement led to an interesting discussion on InfoQ, which included topics such as which applications are appropriate for the cloud and issues of vendor lock-in:

Floyd Marinescu gives his perspective on how this will impact that next generation of application platforms:

I think Google App Engine is the beginning of a whole new category of cloud computing offerings, making the total set in my view loosely similar to the following:

1. Grid or Master/Worker implementation clusters. This is the traditional view of cloud computing, where you just have a master/work type programming model with the workers being executed transparently across a grid of computers. This type of stuff is happening behind the firewall, I don't know of any internet/publically exposed services that do this.
2. The internet becoming a new middleware platform. This is where Amazon, Microsoft, Yahoo, and are playing. Middleware api's/products that previously were installed in the datacenter and charged on a license fee basis are now moving online with web service accessesible API's, charged on a utility computing model. Things like compute (Amazon EC2), Storage (S3, Microsoft SQL Data Services), Queing, domain-specific data sources (the realm of mashups and RSS), are part of this 'trend'. The internet is starting to provide middleware building blocks that are mashed up to build applications. Users can see massive cost savings compared to buying similar tools and maintaining them themselves in the data center.
3. The internet as an application hosting platform.So whereas the previous trend is about exposing middleware constructs on the net, this trend is about exposing a whole end-to-end application stack with API's for everything from MVC/web to messaging to data storage (entities) built in. As a developer now you don't even need to think about things such as scalability, about data storage format, etc. You just deploy your app to this 'platform cloud' and it just works. Google's cloud computing offering is thus a higher-up-the stack offering compared to Amazon.

Galen Gruman and Eric Knorr give a good review of cloud computing in a recent Infoworld article, entitled What cloud computing really means, in which they categorize this type of offering as Platform as a Service:  

Platform as a Service: Another SaaS variation, this form of cloud computing delivers development environments as a service. You build your own applications that run on the provider's infrastructure and are delivered to your users via the Internet from the provider's servers. Like Legos, these services are constrained by the vendor's design and capabilities, so you don't get complete freedom, but you do get predictability and pre-integration. Prime examples include Salesforce.com's Force.com and Coghead. For extremely lightweight development, cloud-based mashup platforms abound, such as Yahoo Pipes or Dapper.net.

What's interesting about all this is the gap that still exists between the Platform as a Service delivered by cloud providers and enterprise application developers. It is clear that neither Google nor Amazon thought deeply about what it means to run existing applications on the cloud, especially those already written with JEE or .Net. What's also interesting is the assumptions they are making with regards to the trade-offs among consistency, performance and reliability. For example, if you look at Amazon's SimpleDB service, the Amazon message queue and other web services, it is clear that they were designed to suit a certain type of application, designed in a very specific way..

In addition, and looking at Google's offering especially, it seems that they came up with their own way of doing things and are essentially dictating that way to anyone who wants to use their platform. As Jason Carriera commented in about Google (in comparison to Amazon):

I can't understand who'd want this... With Amazon Web Services, you get a VM hosting service where you can run any AMI

So while Amazon is still more open than Google, if you want to take full advantage of their Platfrom as a Service offerings, you are pretty much in the same boat.

This brings up some very serious questions:

• If we want to take advantage of one of the clouds, are we doomed to be locked-in for life?
• Must we re-write our existing applications to use the cloud?
• Do we need to learn a brand new technology or language for the cloud?

Geoffrey Wiseman's comment on the InfoQ thread highlights some of the common concerns:

..that scares me precisely because of the vendor-lock-in. I mean, I wouldn't necessarily be shocked to have an ISP reserve that kind of right -- only I would retain the option of hosting it somewhere else. With GAE [Google App Engine], if Google decides they want to take down your app, what's your next option?...

Garett Rogers published his view on ZDnet The problem with Google App Engine, in which he outlined similar risks to those described by Geoffrey:

• You are putting your application in Google’s hands

Think about that for a minute. You are at the mercy of Google — if disaster strikes and Google one day disappears, you are done to. Or, more realistically, if the Google App Engine goes down for an hour, you are also down for an hour — and you will have no idea what happened.

• It’s free right?

Not only are you locked in, you are completely at the mercy of Google’s future pricing strategy for the Google App Engine…

• Privacy should not be taken lightly

     Google has a very strong privacy policy — and personally I trust them. However, I’m trusting them with my personal information — you will be trusting them with all of your company’s data?...

• Once you are in, you are really in

      Using Google’s infrastructure is very tempting. But any smart company should have some sort of plan for the future…

It is clear that cloud computing is still in its early stages and it is, therefore, hard to tell how it's going to shape up. It's also very likely that we will see increasing competition among the cloud hosting providers which means that the last thing we want to do is lock ourselves to a specific provider's platform.

So how do we go about that?

The most important approach I would recommend is abstraction: write your code in a way that is agnostic to the environment in which it will be running. This may sound trivial, but it's not that simple. Following are  some practical suggestions for a practical way of doing this:

• Use development frameworks, such as Spring, Hibernate and Mule, to abstract your application from the underlying runtime implementation. Interestingly, now that Java has built-in support for scripting  languages you can also write pieces of your code in any of those languages, even keeping yourself abstracted from the use of Java.
• Use the runtime platforms of your choice to deliver messaging, parallel processing and data services based on cost/value. At the end of the day, it's the runtime platform's responsibility to deliver its services in a virtualized/cloud environment and abstract the complexity involved from the application using it. 
• Use the cloud of your choice. A cloud could be anything from your local PC to your corporate data center to a public Internet cloud such as Amazon EC2. 

After applying all of these principles, here's how your new application stack will look like:

Note that with this approach, the development framework doesn't even have to be bound to Java, .Net, C++ or scripting. It can combine all of them together.

This sounds a bit complicated -- if you are assuming you need to do it yourself, but the good news is that there is built-in motivation among the various players in that stack to play well together to ensure interoperability, without needing an external "guard" in the form of standards bodies. For example, the makers of Spring and Mule are motivated to support various runtime implementations, and therefore, do a good job creating the abstraction. In turn, the runtime platform providers, such as GigaSpaces, Oracle, IBM or Terracotta, are motivated to support the various development frameworks.

This type of abstraction already exists with various products in the marketplace. For example, GigaSpaces supports Spring, Mule and Hibernate for Java, as well as nHibernate and WCF for .Net.  We're seeing more platforms becoming a part of this ecosystem, such as Compass/Lucene and other projects in  our own OpenSpaces.org such as JPA adaptor, Memcached, SimpleDB and PHP integration (I'll be writing a separate post just about that).

Using that approach we can build our application today in a way that "future-proofs" it, and does not lock us in to a particular platform.

The different solutions compete on various aspects:

• Runtime capabilities, such as scalability, performance and reliability
• How well they provide an integrated solution with the different development platforms
• How well they support and integrate with internal and external clouds
•  How simple is it to develop and deploy an application

The result will be the next generation middleware platform, or Application Server. And IMO, Java, some aspects of JEE (API), and .Net will all play a part in this offering.

The good news with this approach is that if you already have existing applications written in Java or .Net, you can still benefit from the cloud, without needing to re-write your entire application or learn new technologies and languages.

I expect that at some point in the near future some of the cloud providers will support this type of model. By running the GigaSpaces Amazon Machine Images on Amazon EC2, you can already implement this approach today.

One requirement - so many prespectives

Got this picture from a colleague - The picture below shows the different perspectives in a typical sales cycle by the different stakeholders for the same customer requirement.  If it wasn't so true:)