Nati Shalom's Blog

Data agregation pattern for effective monitoring

In my previous post I wrote about two patterns for using a GigaSpaces cluster to solve some of the issues involved in managing distributed applications:

• Using the space as a scalable alternative to a directory service. With this approach each service publishes itself to the space directory service and a JMX proxy acts as a wrapper that virtualizes the different managed services from the client accessing them.
• Using the space as a management repository aggregating management data.
  

Steve Colwill from PSJ published a second post on this topic titled Data Aggregation via JMX and the Grid that covers the second option outlined above.  In summary, Steve suggests that each managed agent will report its aggregate statistics to the space. A JMX façade is used to expose the statistics through a standard JMX API as described in the diagram below:

If you think about it, this is yet another proof of the old aphorism by Butler Lampson that "All problems in computer science can be solved by another level of indirection".

The thing about indirection though is that it basically moves the problem from one layer (the application layer or management layer in our specific case) to another layer (infrastructure layer).  Now if that infrastructure layer is not in place, we have to create it ourselves, which makes Butler's statement kind of empty. This is where Space-Based Architecture becomes handy. It serves as a general purpose infrastructure layer that provide a means for solving data distribution, high availability and scalability. In this case we applied the abstraction principle as a way to expose the generic space capabilities through a specific set of APIs (JMX in this case). The combination of the two creates what I often refer to as a middleware virtualization layer. We use the same API, but the implementation of this API is virtualized. In this specific case, our JMX API doesn't point to a physical JMX server, instead it points to a virtualized cloud of servers.

Quoting Steve: 

One of the reasons I'm a fan of GigaSpaces and space-based architectures is that a number of architectural choices that are traditionally hard-wired: transactional/non-transactional, sync or async replication can be changed through configuration only. This enables common design patterns (and therefore components) to be applied to a wide range of application problems, by enabling the data integrity/performance equation to be tweaked at a late stage of application assembly.

 

Summary

The main benefit of this approach compared to the one described in the previous post is that it decouples the client and the managed services. A client doesn't need to maintain direct a connection to the managed service. Most of the logic is kept server-side. In this way we can keep the client-side --  which acts as the management façade -- stateless and thin.

Because the space acts as a distributed data-store, it is easier to build aggregated statistics, as all the statistics are placed in one logical entity - the space.

Decoupling enables us to easily add new services/policies to our system without changing the management service. For example, we can add SLA monitors that can listen to the statistics in the space and take action once a certain threshold is breached. This capability makes the space an ideal solution for those looking to build their next-generation management and monitoring application.

Cloud management frameworks (for Infrastructure-as-a-Service) are ideal candidates for this, as they have the need for proximity of the management information and application behavior. The management layer acts as a loopback mechanism that can tell the application how it is actually behaving. The application can use this information to adjust itself to meet a given SLA without human intervention.

Having said all that, it is also important to note that the two options presented in Steve's posts are not necessarily mutually exclusive. I could easily see how one could use the approach presented in this post for maintaining the managed dashboard information, and the approach in the previous post to invoke specific operations on a managed service, in case you want to drill down to the managed service level. Having one underlying technology that can be used to serve the invocation, virtualization and data virtualization is yet another benefit of using the space.

Managing application on the cloud using a JMX Fabric

One of the challenges of managing application in a distributed environment such as Cloud/Grid is that collecting or finding the management information of each part of the application is a relatively complex task.

JMX provides a standard way to expose the management information (MBean) of a particular server. However, the way the client-side finds all the MBeans that comprise the application, or the way a single client might interact with the distributed parts of the application, is left open.

Steve Colwill from PSJ wrote a detailed blog, JMX for Grid Based Applications,
where he outlines a solution that uses JMX JSR-160 connectors and GigaSpaces to create a JMX Fabric. According to the proposed solution, the managed agent (server side) use the connector to add a reference of each MBean stub to the space. The client uses a FederatedMBeanServerConnection class that picks up those references from the space, connects to them and then delegates operations to the set of Mbean servers, effectively acting as a multiplexer.

Using the space as a JMX directory service

The above diagram illustrates how the model described by Steve works. The client is abstracted from the physical location of each server and can easily discover services that join the network. The connection from the client to the servers uses peer-to-peer communication, which means that once the service is discovered, no additional overhead is needed for communication between the client and the managed service. In this case, the space is used as a directory service. We leverage the fact that it can be distributed and dynamically discovered to simplify the discovery process in a distributed environment.

Using the space as a management data repository

The above model is quite useful for cases in which we want to expose federated services which have an existing remote interface. But this is not always the case. If it isn't, we can use the space as a management data repository, which contains full management information for each agent and exposes that information to the client or to any management application. In this method too, the client application is abstracted from the managed service. But unlike the first option, the client gets the information about the managed entity directly from the space, and doesn't need to maintain a connection with the managed service. The space in this case is used as a distributed database, so the application can not only obtain management information about an individual server but can also gather aggregate statistics and perform other aggregate data queries, directly on the data model.

Summary

Steve's solution to managing application in a distributed environment is an interesting one, as it enables applications that are already using a standard JMX interface to use a new federated model without changing the application and without adding a performance overhead. This is achieved just by plugging a new space-based connector. It is a good example that shows how a space can be used as a distributed directory service. It is important to note this is only one pattern in which a space can be used to solve this type of challenge. There are other ways; using the space as a management data repository, as I suggested in this post, is just one of them. The nice thing is that implementing any of these patterns becomes fairly simple once the space is brought into the picture.

I would like to end this post by thanking Steve specifically and PSJ in general for being a great partner for such a long period of time, and for sharing your experience in such meticulous detail.