Nati Shalom's Blog

Is Your Data Really Secured?

Security has always been one of those topics that we as developers or architects hate to deal with. Our ideal world would be that security is dealt with at some higher level in our data-center, and that we don't need to think about it. It turns out that some of the most recent frauds can be caused by someone from within our own organization. In this specific fraud case, the cost to SocGen was $7.7b.  There was nothing specific in this incident that was a direct result of an application security failure, but it does force us to examine more carefully some of the current assumptions about security. In this post I focus on some of the potential security vulnerability that we may be exposed to as a result of using data-grid and caching more and more within our organization. 

Why Do We Need a Secured Data-Grid?

We add the data-grid/caching layer as a front end to the database to address the database scalability limitation. One of the common assumptions in this sort of model, is that the caching layer is an embedded part of a particular application. In other words, the caching layer can only be accessed by a specific application. Based on that assumption, it is fair to assume that it is the application's responsibility from a security perspective, to control access to its caching layer. Therefore, security was not generally considered to be a critical aspect of many of the existing caching solutions.

However, there are two things that change that assumption:

• Caching becomes a shared service (just like databases of today), that is shared by more than one application. SOA-based applications are a good example of this. SaaS-based applications impose an even bigger challenge in this regard.
• Virtualization and the drive for better utilization, forces us to think of a more efficient deployment model through sharing of resources. Having a dedicated in-memory cluster per application can be a fairly inefficient model, but on the other hand, running in a shared environment requires a high degree of isolation support by the data-grid service.

Security assumptions and constraints change significantly when you move from a cluster dedicated to one application, to a cluster shared between several applications. It is no longer enough to assume that the application is responsible for granting access to its embedded middleware components. In this case, multiple applications which share the same data, must have the right level of isolation and access. Securing the shared middleware resources is the key to achieving this level of isolation and control.

In the next section I describe the specific attributes that a secured data-grid must support.

Securing Your Shared Data-Grid/Caching Layer

1. Granting Access Based on Roles and Content

Imagine the following cases:

• Several applications can access the same data-grid service, and be granted different permissions to view each data item, based on the content of the data.
• Different parts of the same application (e.g. purchasing module vs. sales module) may be granted access to different accounts, even though all the account data is stored in the same shared data-grid.
• One part of an application can be granted read/write access, while another part of an application can be granted read-only access to the same data.

This scenario is very similar to what we expect from most databases today. A user can be granted access (authentication) through user-name/password authentication, and they can be granted read/write access (authorization) to the shared data. Providing such a level of control is fairly basic and core to any secured data-grid. However, there are cases where we would want to grant access to different parts of organizations, or where customers need to have different privileges for different parts of the shared data. For example, in a call center application, we might have a shared call center service that serves different departments in our organization. In such a case, we would want callers of certain departments to see only the contacts that belong to them, and not be granted privileges to see or have access to contacts that belong to other departments. In that case we would need to provide fine-grained authorization, based on the actual content.

2. Ensuring Isolation in a Multi–Tenant Environment

There are many cases where we do not want to share the same data-grid between different groups of applications, but at the same time we do want to share the physical memory resources. A multi-tenant environment implies that we run multiple services or applications in a shared environment, but keep them logically separated, as if they were running in their own dedicated environment. That level of separation is referred to as isolation. In order to do that, we need an abstraction layer that enables us to provision our data-grid as a set of fine-grained services that can be managed independently, even if they run on the same process. Abstracting the data-grid as fine-grained services enables us to provision the data-grid services on a shared group of machines or even processes (JVM’s), and still manage them completely independently.

There are a few components that enable that level of fine-grained isolation:

• Life cycle management – each data-grid can be started or relocated, completely independently of other data-grid services, even if they run on the same process container.
• Class-loader  - each data grid contains its own class-loader to ensure that there is no class version conflict between two data-grid services.

In addition to that, there are other levels of isolation, such as ‘zones’ where we can ensure that certain data-grid services never share the same process or even the same machine. At the top level, we can assign a data-grid with specific virtual machines (VM), which ensures the highest level of isolation, not just at the service level, but also at the operating and physical resources level.

Providing this level of fine-grained isolation control between data-grid services that share the same process, up to a completely separate data-grid that runs on its own virtual machine, is critical if we want to ensure best utilization on the one hand, and isolation on the other hand. Providing a limited degree of isolation determines how effectively we can share resources.

3. Transport Level Security

In most data-grid deployment, it is fair to assume that network level security is a responsibility of the data-center, not the data-grid, or even the application. There are cases where the data-grid can span across multiple firewall or data-center zones, and have a replication channel between those zones. In such cases, the replication channel can turn out to be a security loophole. The simplest approach to address this challenge, is enable replication over SSL. Using SSL can ensure that no one can “sniff” the packets that go over the wire.

The Administrative Challenge

Providing all this level of fine-grained security control and multi-tenancy can pose a huge administrative challenge. For example:

1. A super administrator needs to have full access to all data-grid services

2. An application administrator only needs to be granted access to data-grid services that belong to a particular application.

3. Operational managers need to be granted monitoring (read-only) access, to view how the system is behaving, but don't need to be granted access to change the data-grid topology or configuration.

These are only few of the challenges that we have to deal with, once we move from a dedicated data-grid to a shared data-grid. This basically means that we can’t assume that all administration operations to our data-grid are the same.  We need to treat operation guys differently then we deal with application managers, and they all need to have visibility and some level of control to our shared services. That means that we have to rethink how to manage security when we deal with shared data-grid services.

Not Just Your Data..

The same arguments that I have mentioned here, apply not just to the data-grid, but to any part of the application middleware that is going to become a shared resource. For example, think of the messaging service, or any other part of your application that becomes a service to other applications. This implies that we need some sort of generic way to handle application security that is consistent across the layers of the application.

How Security Reduces Costs

Being able to share data-grid services between multiple applications is a great cost saver. Here is how:

1. Better utilization  - we can better utilize the machines that run our data. In other words, we reduce the total number of machines that run our data services. The fact that we use less machines to run our data-grid services means that can save all of the cost associated with it i.e. cooling, maintanance, space etc.

2. Pooling of spare resources – each data–grid needs to run some level of spare capacity. Sharing of data-grid services enables us to share that spare capacity, and therefore better utilize it amongst different applications.

3. Reduction of the amount of redundant data – one of the ways that organizations dealt with the lack of ability to share their data services effectively, was by maintaining different copies of that same data. That led to redundant copies, as well as a lot of synchronization cost associated with it. Being able to safely share the data, means that we remove a lot of that overhead, and the cost associated with it.

4. Reduced administrative overhead – the data-grid becomes just like a data-base: instead of having each application group learn how to provision it separately, and go through the cycles of tuning and sizing it separately, we now have all that knowledge and skill set managed by a shared group. This is the same way in which Amazon manages its SimpleDB, SQS and other services - we as users of those services don’t need to worry about how to provision or maintain it.

Final Words

Caching/data-grids are going through a similar evolution to databases. As with databases, we started by using caching as an embedded service to the application. Now we are in the phase where we need to be able to share the data between multiple applications, or in cases where we don’t want to share the data, we need to be able to share the resources for managing the data, while keeping a high degree of isolation. The demand for these sort of requirements becomes much more common with SOA or SaaS-based applications.

We used to think of security as a defensive move. However, security is not just about protecting our data – security can create the right level of isolation that is a main enabler for achieving better efficiency and cost savings, through sharing of resources, as I pointed out above.

As we approach the next generation of middleware and data-centers, it becomes clear that we cannot move to the next wave of virtualization and cloud computing without a strong security and isolation solution that is built-in to all layers of our application and middleware.

GigaSpaces Security as a Reference for a Secured Data-Grid:

If you are interested to learn how these challenges have been addressed specifically by GigaSpaces, you are welcome to join a technical webinar that we will be conducting on that subject. The webinar will take place on Wednesday, October 28, 2009 9:00 AM - 10:00 AM PDT

GigaSpaces and GoGrid: Cloud Platform for Enterprise Applications

I had few discussions in the past week with people who asked me about our recent GoGrid announcement and how our cloud offering integrates with the GoGrid environment. I’ll try to answer the most frequently asked questions below. In addition, we’re conducting a webinar where you can ask your specific questions online and watch a live demo – see details at the bottom of this post.

What is the GigaSpaces cloud offering?

GigaSpaces provides a set of cloud-enabled middleware services, namely data grid, messaging and Map/Reduce. These are equivalent to the Amazon SimpleDB, SQS and Map/Reduce, with the the main difference being that it’s implemented entirely in-memory and designed to support existing enterprise applications. On top of that GigaSpaces provides integration of those services into a Platform as a Service solution designed to make the deployment of JEE/Spring or XAP-based applications extremely simple.

What makes it enterprise ready?

Most enterprise applications are based on standard JEE or .Net platforms.  In addition, all enterprise applications are transactional in nature and cannot tolerate any potential inconsistency. Enterprise applications are more sensitive to performance and latency. Our cloud-enabled middleware supports all these requirements. It was designed to leverage memory resources to gain the best performance and latency, and supports large-scale deployments from the get-go. It is based on XAP, which is a mature product with hundreds of deployments in the most mission-critical applications.

What makes your middleware cloud-enabled?

GigaSpaces supports all the key characteristics expected from cloud-enabled middleware, as defined by Gartner below:

How is it different than traditional middleware?

With GigaSpaces all middleware services run on a shared pool of resources. GigaSpaces provides a high level of security to ensure complete isolation between multiple applications. The deployment, provisioning and monitoring is completely automated.

Can your middleware seamlessly shrink and grow dynamically when a machine joins or leave the network?

Yes – let me explain:

There are three components that makes our middleware elastic and thus suitable for a dynamic environment:

1. Service automation and administration – this part takes care of moving the middleware and application services around based on machine, memory, and CPU utilization.
2. Common clustering and state management – this part makes sure that nothing breaks when you start to move around any of the applications or middleware services.
3. API Facade – this part provides a standard interface as well as language binding, for seamless access from existing applications.

What does the integration with GoGrid include?

The integration includes the ability to use any of the middleware services (data-grid, messaging and Map/Reduce), either individually or integrated with our PaaS solution.

Can I use it if I'm not running on GigaSpaces XAP?

Yes. Users can leverage the GigaSpaces images to get deployment automation and improved management for their existing application, as well as add auto scaling and high availability. 

What is unique about the GoGrid cloud offering?

For GigaSpaces users, the main things that you’ll find useful in the GoGrid environment are as follows:

• Enable custom deployment – GoGrid enables enterprise users to customize their environment to fit their specific enterprise standard. This include VPN, network as well as machine hardware configuration if required.
• Native multicast support – makes cluster deployment simple.
• Standard database storage support – makes database high availability simple.

For GigaSpaces windows/.Net users, you’ll find the following useful:

• Built-in .Net support
• Support for Windows and SQL Server 2008
• GoGrid have the same pricing for 64/32 bit

When can I try the GoGrid integration?

The GigaSpaces/GoGrid solution is currently available for limited set of users. Public availability is expected within a month. If you’re interested in early access, or want to be notified when the images will be publicly available, please send an email to <cloud at gigaspaces dot com>.

Is there a live demonstration?

To see a live demonstration, you should register to the Enterprise-Grade Cloud Platform with XAP and GoGrid webinar that will take place on October 14 at 9:00 AM PDT.

Why Larry Don't Get It

Once again, Larry Ellison, CEO of Oracle, made a joke of Cloud computing, saying that "a cloud is water vapor."

Ellison made the same argument last year – Geve Perry wrote a nice piece on Ellison's Anti-Cloud Computing Rant back in September 28, 2008 (note that the timing in both cases happens to be fairly close to the Oracle World event).

Many of the arguments that Geva brought back then still apply today. Clearly, Larry has a vested interest to argue that “nothing is new under the Sun.”

I actually tend to agree with Larry – Cloud doesn’t bring any real new technology to the market.

But what Larry, and many folks that he represents, don’t get is that cloud is not about bringing a new technology to the market. Cloud is about making the the same concept that we have today much more efficient and simple so that it can be adopted by the masses. Everyone in this business knows that building technology for high-end enterprises is vastly different then building the same technology for small startups. But again, I completely understand why someone who wants to re-brand an old mainframe concept as a new “database machine,” and bought an entire hardware company to serve that purpose, would claim that there is nothing new under the “Sun” (no pun intended).

Anyway, I'll have to give Larry credit for being a real performer and hilarious in the way he articulates his arguments.

For those who missed the video here it is:

How Map/Reduce and the Cloud are Affecting Analytics Applications

Two weeks ago I gave a presentation at the Sun HPCW workshop in beautiful Regensburg, Germany. It was a great chance to meet many of the folks from the Sun Grid Engine team face to face.  It was also a great chance to take a step back and look at the evolution of one of the interesting areas in the industry - analytics. Even though this is one of the more technically challenging areas, I find it odd that relatively little has been discussed about this within the developers community. 

In this post I will try to provide an overview of the different approaches for running analytic applications. I will also examine how patterns like Map/Reduce and a new environment such as the Cloud, drive a new approach that is significantly simpler and more cost effective than many of the existing approaches. There are other models for analytics such as stream based processing or CEP that i chose not to cover in this post as they deserve represent a new paradigm that deserves a separate discussion.

What is Real-Time Analytics (a layman’s definition)?

Here is the simplest and shortest definition I could come up with (probably not the most accurate one, but good still good enough):

· Analytics: The process of manipulating raw data (a.k.a data mining) into a meaningful report.

· Real-Time Analytics: The ability to generate the desired report in near real time.

Analytics in the Finance Industry

It is interesting to note that financial applications within organizations are organized based on the “speed” with which they need to react to the market. The Front-Office for example, would be the trading/market data application that needs to react to the market event in a fraction of a millisecond. The Middle and Back-office is the part of the organization that looks at an aggregated view of the market and tries to draw various trends, based on the current market and history. The algorithm that is used to process that data and generate the required report is referred to as Monte Carlo simulation.   A typical set of reports generated in that model would be P&L (Profit & Loss), VAR (Value at Risk), Reconciliation etc.

In previous years, those reports were generated on a post trade basis, i.e. at the end of the trading day, the data from the current day was aggregated into files or a database, and during the night there was a set of services that processed the data and generated the desired reports. That meant that if there was an event during the trading day that could affect the portfolio, it would only be taken into account the next day. At the same time that the volume of data increased, the window of time to process that data (during the night) became shorter due to globalization (time zone differences shortened the “night”). This led to scenarios where simulation accuracy was limited by these margins.

Analytics Becomes Mainstream

In general you can categorize any application that looks at an aggregated view of a stream of data, as an analytic application. Fraud detection would probably be another good example of analytics, since it tries to continuously monitor user actions in order to determine whether their action is part of a fraud or not. Social networks, Ecommerce and even personalization, such as search engine personalization have become more mainstream. With that, the demand to process vast amounts of data to produce various market trends, user behavior, fraud behavior etc. becomes not just useful, but critical to the success of the business. A search engine would be a good example of that – the quality of the search results is a direct function of how well you are able to aggregate content and user search patterns, and apply it to the way you produce the search results.

The Evolution of Analytic Application Architecture

To make things simple, I will start with a very basic view of the analytical application architecture, which represents the two most common architectures:

Phase 1: First-Generation Analytical Applications

Phase 1-1: Excel spreadsheet

Everyone uses Excel to generate various reports, so the use of Excel and spreadsheets are probably the most common and popular way of generating reports. The way this is done is by loading the data from a file, and then using Excel to generate the graphs and various views of that data.

Phase 1-2: High-End ETL process

Excel works well for small data sets, but for large data sets a batch process is used that is based on ETL (Extract Transfer and Load). It represents a batch process in which the operational data from the production sites is normally aggregated into some sort of data warehouse. In the process of doing so, the data is stored in a format that is optimized for generating reports. The process normally involves extracting the data, transferring it into an optimized format, and loading it into the warehouse for further processing. The ETL process normally happens once per data source. The data processing (a.k.a data mining) is where most of the heavy duty work takes place. In many cases, once the relevant reports and analysis have been produced, the data becomes obsolete and at this point it is common to clear it and store it in a long term archive.

The Need for Real-Time Analytics

You don’t need to be a genius to realize that that if all reports could be generated in real time, this would probably be the ideal world. The only reason this is not done, is purely technical. No way had been found to aggregate all this vast amount of data and process it in real time. Traditional databases had to be tuned completely differently, to deal with real time transaction processing and complex analytics (type of locking, data structure, transactional semantics etc.).

“Real-time data … requires a continuous flow of data, and traditional enterprise data warehouse deployments introduce slow data delivery to BI environments” (Forrester)

The challenge became even worse as the volume of data increased exponentially, and at the same time there was a continuous pressure by businesses to process even more complex analysis at an even lower cost. It was clear that addressing those contracting requirements couldn’t be achieved by just “throwing more hardware” into the problem. That led to the next phase of analytic application architecture, i.e. a grid-based analytic system.

Phase 2: Second-Generation Grid-Based Analytical Applications

Phase 2-1: Speeding up Analytics via Parallel Processing

Phase 2 was not yet trying to get into real-time analytics but was mainly trying to find ways to speed up the process of analytics. The majority of the focus in that area was on the batch processing that in some cases could take days or hours. By introducing grid computing, the processing time could be brought down from days to hours, and from hours to minutes.

With grid computing, the way that processing time is speeded up, is simply by spreading the processing into a large pool of commodity hardware. By splitting a linear process into small chunks that can be processed in parallel, the processing time can be speeded up quite significantly.

Phase 2-2: Adding an In-Memory Data Grid

As the time to process the data went down from hours to minutes, other factors that were previously considered insignificant became much more significant. It became clear that to speed up the analytics beyond the current level, closer to real time, it was necessary to consider ways to speed up the load and data access time, because they became more significant relative to the processing time.

“Real time drives database virtualization - Database virtualization will enable real-time business intelligence through a memory grid that permeates an infrastructure at all levels" (Forrester)

This approach became fairly common during the past few years. There is more information about this approach in one of my previous posts:  Bringing Data Awareness to the Grid

Challenges with Grid-Based Analytical Systems

While technically the combination of Compute and Data Grid were able to bring the time it takes to perform even very complex analytics to near real-time, the adoption of that solution was still marginal. One of the main reasons that this solution was not widely adopted was cost and complexity. Only a few organizations could afford building and maintaining large data centers. The skill set for running and building applications that could run in those environments was fairly rare, and therefore the labor cost was also rather high.

Phase 3: Next-Generation Public/Private Cloud-Based Analytics

The cloud makes a grid-based system affordable. So with the cloud it is possible to reduce the hardware cost, and more importantly it avoids the huge initial investment. The fact that the cloud is more widely adopted also reduces the labor cost and ramp-up time required to build such systems.

Having said that, the fact that anyone, even the smallest startup can have a large data center at a distance, reduces not just the initial investment and the direct hardware cost, but opens up a new set of opportunities for building such applications. For example it is possible to outsource the entire analytic process to someone else and use it as a service.

This was the case with Primatics Financial that provides risk analysis for mortgage portfolios as a service.

The Cloud and Map/Reduce Make Analytics Simple

With grid-based systems that run in the data center, the pool of machines is fixed. One of the main roles of the grid scheduler is to manage the pool of physical resources which belong to that pool. The cloud on the other hand already takes care of most of the resource scheduling part. Virtualization makes it possible to get the isolation required to maintain the separation between multiple jobs. If these two factors can be leveraged, a simpler model can be developed.

One of the ways to simplify the analytic application is to simplify the way that parallelization is handled. Since most of the resource scheduling is dealt with through the Cloud/Virtualization layer, a simple pattern such as Master/Worker, Map/Reduce can be used to parallelize, i.e. to distribute the application across the available machines.

You can read more about the Map/Reduce, Master/Worker approach here, and how it compares with a grid-based system.

Reducing the Number of Moving Parts

Another way to simplify and optimize the application is to reduce the number of moving parts.

Rather than having a different set of services to load the data, hold it in memory, and yet another set of services that computes the analytics, it is possible to bundle the loader, the data and the processing services into one application unit, and have multiple instances of those units running to handle the scale. In this case, each unit handles the entire life cycle (the loading, processing etc.) of the data analytics food chain.

SaaS Enablement through Multitenancy Support

One of the core elements that is required to provide the analytics application as a service is multitenancy. Multitenancy is the ability to run the multiple analytics processes on a shared infrastructure as if they were running on dedicated resources. To add multitenancy to our service, the life cycle (Start, Stop etc.) of each of the analytics needs to be managed independently of the other analytical processes that are running over the network.

There are two possible layers of isolation that can be chosen:

Service-Level Isolation

Service-level isolation provides isolation at the application component level. You can deploy multiple services on the same process/machine; you can run different versions of that same service in the shared environment and maintain them in complete isolation to other services that are running in the same environment.

Since all services share the same machine, they can potentially access shared resources such as the file system, etc. It is also harder to control things like CPU utilization and memory allocation on a per service level.

Machine-Level Isolation

Machine-level isolation is provided through the use of a virtual machine. In this case, each application or group of applications runs on separate virtual machines. Virtual machines provide an additional level of isolation that includes the actual CPU, Disk and Memory resources.

Combining Service-Level and Machine-Level Isolation

Both service-level and machine-level isolation are actually complementary approaches, as each one of them provide a different level of isolation and application packaging granularity. Service-level virtualization tends to be more fine grained, but at the same time provides a lower level of isolation. Therefore, it makes sense to use it in areas where real CPU, Memory and file system isolation are not that critical, but getting better utilization from the existing machines is more important, and vice-versa.

In the diagram above, you can choose to run everything from the same organization at a service level of isolation, and use a VM level of isolation between different customers to ensure that each customer’s application is running in a completely private set of VMs.

Private/Public Cloud

With the introduction of private cloud providers such as VMware with their Vcloud and Vsphere approach, and open source alternatives such as Eucalyptus, you can choose to run the same solution in a hosted environment or locally. You can also use a hybrid approach where some of the processing runs in your local environment and the rest in the hosted environment. Things like the Amazon Virtual Private cloud make such a hybrid deployment model fairly simple, as the local and hosted machine are accessed in the exact same way.

Final Words

The continued demand for processing a growing amount of data in real-time and at a lower cost, drives continuous innovation in the way data is processed.  Grid-based analytics (Compute and Data) addressed the demand for scaling, and also reduced processing time quite significantly, but at a relatively high cost and complexity. The demand for real time analytics by mainstream applications in the social networking media created a drive for a simpler and less costly solution. Cloud computing enables the cost of hardware to be reduced, but at the same time opens up new opportunities for making the analytic processing significantly simpler. You can rely on the cloud for scheduling and managing your resources, and on virtualization for multitenancy. With that, a simplified parallel processing approach, such as Map/Reduce, can be used to handle the parallelization of the application.

“it's quite clear that the business-intelligence/data-warehouse industry is moving toward a new paradigm wherein the optimal data-persistence model will be provisioned automatically to each node based on its deployment role -- and in which data will be written to whatever blend of virtualized memory and disk best suits applications' real-time requirements." (Forrester)