Discussions

News: Tim Bray on grids

  1. Tim Bray on grids (16 messages)

    Tim Bray, in "On Grids," offers "an introduction to the state of the art in Grid Computing in mid-2006. I start with some definitions and motivation, look at grid economics, survey the kinds of infrastructure that are out there now, and touch on future directions." It's a very interesting and informative read.
    Houston, we have a problem: nobody agrees what "Grid" means. Check out Wikipedia for a list of alternative definitions. Whatever. I think that the massed Opterons folding proteins in university computing centers are grids. I think the Google and Yahoo data centers handling a kazillion searches a day are too. And I think that SETI@Home is too. So I’m going to be sloppy and use “grid” to mean any scenario where you want a good way to throw a whole bunch of computers in parallel at a problem, and you want some infrastructure there to help.
    Mr. Bray points out where grids are great ("One reason is that we’re all generally moving toward scaling out rather than up; throwing lots of relatively cheap machines at problems in parallel, rather than trying to use one big honking mainframe-class box") and where they're not ("memory space and compute cycles are pretty cheap. Disk space is effectively free. Moving data around in large quantities is expensive.") He also discusses some implementations of grids (in various languages) and the strengths and weaknesses of the implementations. He's going to be writing more about this, and with grids becoming more important for high-end scalability, the information is good to have. (After all, companies like Terracotta, Gigaspaces, Tangosol, and Gemstone are advocating grids successfully, not to mention Azul and other companies that provide some similar functionality.) To back that up: Nati Shalom, of GigaSpaces, discussed a SOA architecture that used a grid for local communication (i.e., in a satellite installation) and an ESB for communication to centralized ERP systems or to other satellites, for very high performance in the satellite.
  2. T-->G and B-->G[ Go to top ]

    Tim Bray and Jim Gray - transform: T-->G and B-->G :)
  3. It occurs to me that earth-bound architects may miss the point that the JavaSpaces specification has in no way any dependencies on the use of orbiting satellites or other actual "space-based" constructs. Rather the concept of a "Space-based architecture" is one that promotes using a single logical resource to share: *Data *Logic *Events --where all of these things are simply Objects. Services then: *Interact with each other through the space *Can be co-located with the space for better performance possibilities This means that with a JavaSpace as your hub and services on the "grid" as your spokes, you can indeed build orbiting applications where the center of gravity is the space implementation. --OK, so I may be having too much fun, but I have had a hard week so forgive me. : ) Cheers, Owen.
  4. "To back that up: Nati Shalom, of GigaSpaces, discussed a SOA architecture that used a grid for local communication (i.e., in a satellite installation) and an ESB for communication to centralized ERP systems or to other satellites, for very high performance in the satellite." So far most applications associated with Grid was the classical batch processing type of applications i.e. taking stateless batch processing and parallelize them over commodity HW to improve utilization, reduce processing time and cost. Most of the business critical applications however, are stateful, high throughput applications. The main challenge that were facing is how to bring those applications into the grid. Google is a good example for using grid based architecture for scaling out high throughput application that needs to handle huge amount of data in real time. Key elements in their to enable their scalability were the use of in-memory data-grid, data partitioning and parallel processing. While this model had proven to be successful for Google, most organization cannot afford the level of investment required to build such systems. The solution is to provide all of the three parts of the equation e.g. in-memory data-grid, partitioning and parallel processing capabilities at the middleware layer. In order that the middleware wouldn't become a bottleneck the middleware needs to be ready to run over a dynamic pool of machines transparent to the application. Unlike most of the current middleware solution which are relatively static in nature, middleware built for dynamic grid environment need to be SLA driven i.e. capable of scaling up or down based on load, failure etc. An example for such middleware is provided as part of our Space Based Architecture (SBA). Combined with the POJO based approach through frameworks like Spring and Mule writing business logic on such environment can be made significantly simpler. Using that model almost everyone can run he's application on the grid and scale out the application without changing the code. Nati Shalom CTO GigaSpaces Write Once Scale Anywhere
  5. Nati -
    Google is a good example for using grid based architecture for scaling out high throughput application that needs to handle huge amount of data in real time. Key elements in their to enable their scalability were the use of in-memory data-grid, data partitioning and parallel processing.
    I realize that everyone is amazed by Google (heck - I am), but Google is actually a horrible example for business applications, because correctness (what computer science refers to as "reliability") is simply not a requirement at Google, which allowed them to make all sorts of drastic engineering trade-offs in their architecture and implementaion. As a result, the "Google search grid" is in many ways both completely stateless and completely non-deterministic. As you know, business applications tend to require both stateful and deterministic behavior. ;-) Building "sigrid" may have been a fun exercise for Tim, but business application architects and developers are looking for some pretty specific things, such as: 1) stable state image, even during and after server failure 2) resiliency and consistency of operations across a distributed environment 3) guaranteed semantics, such as "once and only once" and "guaranteed delivery" 4) all or nothing semantics (e.g. no "partial failures") 5) automatonic processing (e.g. no manual recovery of transaction logs, no manual rollforwards and rollbacks) Only after guaranteeing the various necessary qualities of service can one even earn the right to worry about scalability. One cannot later add correctness and deterministic behavior to a system. It is either designed in, or it is not. Regarding the capabilities that were described in the article, the map/reduce capability and much more (but with the guarantees stated above) has been available in our Coherence product for a while now (Jason posted a link), as has the grid-wide batch and service invocation capabilities (since 2003). The truth is that Tim wanted to do something fun, so he made sure that there was nothing out there that would prevent him from deciding to have some fun ;-) Here's the problem, though:
    My problem is that I’ve been a Unix guy for twenty years and a Web guy for ten. My feeling is that if something says it’s a service, the right way to talk to it is to get a hostname/port-number combination, call it up, send a request, and wait for a response.
    Tim is still thinking in 1980s terms. He is still thinking client/server, but he's trying to rename it as a grid. Servers no longer have a hostname/port. In the grid, there is no such thing as a server. All you have is the grid. The servers within the grid are no more individually addressable and reachable than the CPUs within a particular server. If the grid needs to expose a service, it does so through a port on a load balancer that represents the grid as a whole. Apologies to Sun, but: The Grid is the Computer. Peace, Cameron Purdy Tangosol Coherence: The Java Data Grid
  6. Cameron, You are 100% correct with your discussion of the "enterprise grid": The enterprise grid is about state virtualization (which provides guaranteed and predictable resiliency, performance and scalability) - and this is the message we at GigaSpaces (and you at Tangosol) have been trying to promote over the years. One problem we are facing is that the word "grid" means different things to different people. Most people still think of grid as a glorified scheduler.. In fact, grid (as in the utility grid) means resource virtualization. Even the mainframe is a vertical grid to which you can send jobs and get results back. The horizontal grid of 2nd generation (Platform, DataSynapse, etc.) is built mostly around virtualization of processes: i.e. your SAP applications, financial applications etc. - as operating system processes. As the 3rd generation of grids - What we at GigaSpaces and you at Tangosol are trying to do is "embedded virtualization" - i.e. for the first time - the grid is inside the app as opposed to interacting with it from the outside. This is also why we complement as opposed to compete with the traditional grid vendors mentioned above. Gad Barnea GigaSpaces, Inc.
  7. Hi Cameron It's encouraging to see that we've finally moved from a discussion on how to improve performance of J2EE through caching of DB tables to the broader (and more high-impact) architectural challenge of linear scaling-out of stateful applications. Even though your post still talks mostly about pure caching, it's a good sign of market maturity to see that the term Data-Grid, which we've been promoting since 2003-4 is finally starting to catch on :-) I'm sure that having you behind it will help increase awareness around it. However - your focus just on Data-Grids in the context linear scaling-out of stateful applications is not complete and may be misleading. Looking at today’s grid middleware, you would find either - "Compute-Grid middleware": primarily for distributing "batch-type" processing and optimizing the utilization of Data Centers or; - "Data-Grid middleware": for solving the data bottlenecks of distributed applications or J2EE. - I agree that Data-Grid is an essential component in bringing stateful, high throughput applications to the grid (e.g we have demonstrated together with leading HW verndors that you can bring 2 Terabytes of data to the in-memory data-grid and scale reasonably well beyond that.) However - it is not the full picture. To *really* solve the scale-out architecture challenge, one must go beyond the Compute-Grid or Data-Grid dichotomy and embrace end-to-end scalability of both data and processing in an integrated manner. The real trick is how to do it without adding unbearable complexity to the developer. With today's separation between compute and data grids, we leave the integration of the two, and therefore a large part of the complexity, at the feet of the application developers. In my view the solution should be implicit within the middleware layer i.e. we will start to see a new generation of middleware solutions that will be "grid" aware and will have built-in SLA behavior embedded within the middleware stack implementation. Application developers will continue to focus on writing their business logic pretty much in the same way their doing today and rely on that next generation middleware to handle scalability event, failure event dynamic provisioning implicitly". We already provide such solution through our Space-Based Architecture (SBA). With SBA we have a true virtual middleware that supports variety of standard based API on top of a common clustering infrastructure. By means of grid aware I mean that it can dynamically add more instances as needed re-deploy itself between alternate machines in case of failure, move the business logic along with the middleware and manage both the middleware and the business logic associated with it as a single unit of work. This will allow application developers to continue writing their business logic in much in the same way they're doing today, with total ignorance of the underlying physical deployment, and rely on that "grid-aware" middleware to handle implicitly and transparently the performance, scalability, failures and dynamic provisioning. We refer to that vision as "Write Once, Scale Anywhere" and we already provide these capabilities today in many of our customers' production installations. By exposing these capabilities through existing standard middleware APIs such as JDBC, JCache/Map JMS and JavaSpaces, we ease the transition to this new paradigm. Combined with a POJO based approach through Spring and Mule we can now enable J2EE, ESB and a large variety of other applications that didn't even consider grid as a solution, to join the game and become truly high-performance, linearly-scalable, stateful applications with minimum intrusiveness. Nati Shalom CTO GigaSpaces "Write Once Scale Anywhere"
  8. Hi Nati -
    It's encouraging to see that we've finally moved from a discussion on how to improve performance of J2EE through caching of DB tables to the broader (and more high-impact) architectural challenge of linear scaling-out of stateful applications.
    "caching of DB tables" is what TimesTen does. We do not (and never have) cached DB tables. We reliably and coherently manage Java objects in memory across a cluster, starting with the first production deployments of Coherence 1.0 in Q4/2001.
    Even though your post still talks mostly about pure caching, it's a good sign of market maturity to see that the term Data-Grid, which we've been promoting since 2003-4 is finally starting to catch on :-) I'm sure that having you behind it will help increase awareness around it.
    When we introduced linearly-scalable fault-tolerant transactional in-memory data management to the market in 2002, there were no terms to describe it, so we called it "distributed caching". In many applications, these technologies are still used for purposes that I would describe as "caching", i.e. fast data, in memory, with the long-term persistent store in a database or on a mainframe. The difference between old-style caching and what we do is obvious: We manage the real-time, up-to-date data in memory, quite often more up-to-date than what has made it back to the database, which means that we have to manage it in a fault-tolerant and resilient manner (to ensure that it eventually makes it back to the database). That is why these applications are stateful applications -- they are managing their own running state, in memory, across any number of servers. Similarly, Oracle 10g is "just a cache" of a tape backup. ;-) As you and I have discussed, Tangosol only started referring to Coherence as a "data grid" when analysts tried over-generalizing the use of our Coherence software in financial grid infrastructures. They were using the generic term "grid" to describe our software, which we felt was misleading, because Coherence was providing grid services (services to grid applications), but doing so within the grid.
    However - your focus just on Data-Grids in the context linear scaling-out of stateful applications is not complete and may be misleading.
    Perhaps this is where we differ. The only part of "stateful application" that doesn't automatically scale linearly when you run it on 1000 servers is the "state" part. If the application is *not* stateful (for example, an Apache server) then it automatically scales linearly to however many servers you start it on. We understood this problem first-hand (and before building Coherence) from working on over one hundred of the world's largest -- and most challenging -- J2EE applications, including at many of the world's premier financial institutions. As a result, we consider "data as a service" to be the fundamental enabler of scalable performance, and a large enough problem domain that we will easily be able to saturate an entire engineering group's time in R&D for the next dozen years solving just these problems -- and there will still be challenges left at the end of those dozen years! So the "focus" of our Coherence engineering team is hardly accidental, is certainly not misleading, and is (happily) quite profitable.
    Looking at today’s grid middleware, you would find either "Compute-Grid middleware": primarily for distributing "batch-type" processing and optimizing the utilization of Data Centers or "Data-Grid middleware": for solving the data bottlenecks of distributed applications or J2EE.
    The field is already way more rich than this. International banks such as [name witheld] (that deployed the first Coherence financial grid in 2003) already had a rich set of infrastructure for managing and monitoring their infrastructure back in 2003. For their first Coherence deployment, they already had a risk calculation engine and a load distribution infrastructure, and they just needed a data grid onto which to stitch that engine in order to provide the data at a rate fast enough to keep up with the CPUs. Another international bank, [name witheld], built out a major operational grid as a strategic initiative, with all software selected after that point having to run within that infrastructure. In their case, the monitoring software for their grid isn't just for providing sufficient capacity, but is actually responsible for license management (true "pay for what you use" capacity-on-demand). One of the big-four equities firms, [name witheld], got tired of waiting for software companies to deliver, so they built their own grid infrastructure from the ground up. Whatever they buy/build/use has to fit within that infrastructure. Dozens of banks and financial services firms that use our software have made strategic investments in the DataSynapse grid infrastructure, and they will not select any solutions that don't run hand-in-glove with that infrastructure. There never seems to be a shortage of companies out there that want to "be king", i.e. they want to be the company that gets to write the "public static void main(String[] args)" method. They all want to be "the container". They all want to be "the server". They all want to be "the grid". What a giant battle just for the privilege of writing the "main" method! Fortunately, our customers win every time, because we fit perfectly inside each and every one of those containers, servers, grids, etc. As just one example, Coherence-based components built for J2EE are deployed inside WebLogic applications with no changes, and those WebLogic applications are now running inside DataSynapse FabricServer environments -- again without any changes. Peace, Cameron Purdy Tangosol Coherence: The Java Data Grid
  9. Hi Cameron
    "Perhaps this is where we differ. The only part of "stateful application" that doesn't automatically scale linearly when you run it on 1000 servers is the "state" part."
    I agree this is where we differ. In my opinion this is a relatively simplistic view; scalability, latency and performance are usually measured in many benchmarks on a single-tier basis; i.e. messaging or data (mainly due to the complexity of the measurements). However, in reality processing a business transaction involves sending it through a messaging tier (or some other tier), processing it in the business tier and then storing it in a database tier. In addition, each tier needs to maintain its own high availability strategy, which can significantly impact the performance of any given system. To maintain consistency between these separate systems, you'll need to add a distributed transaction coordinator, which becomes another source of complexity, bottlenecks, failure, and so on, and also adds complexity to the deployment environment, all of which can easily bring a system to its knees. This becomes even more complex if you try to apply a dynamic scaling policy in such an environment. In other words, as long as your solution is dependent on other tiers, you'll never be in a position to guarantee end-to-end scalability or even reliability, since you'll only be as strong as your weakest link. And your weakest link, in many cases, will be a part that you do not control. Last week in Java One I met Steve Haines, who was kind enough to share his excellent book. The book is full of scenarios and examples of what I mentioned above. This becomes even more challenging when you deal with scalability. Quoting from the book:
    "Performance vs. scalability: Performance measures the speed with which a single request can be executed, while scalability measures the ability of request to maintain its performance under increasing load". Source: Steve Haines, Pro Java EE 5."
    In my view you cannot guarantee end-to-end scalability, or even performance and latency of a business transaction, if you cannot control the execution of that transaction end-to-end. You cannot perform true dynamic scaling if you cannot pro-actively deal with scaling issues as they happen: if you cannot proactively deal with a failure scenario, except by switching to an already-running instance; and if you cannot apply the same scaling or fail-over policy to the business logic associated with the transaction. And in a large-scale deployment, you cannot guarantee availability without providing proper deployment and management capabilities. You become an enabler, albeit an important one, but just an enabler that leaves much of the complexity in the hands of the application developer. The end result is that you cannot maximize potential performance and scaling gain of the available resources. Having said that, I do agree that in some cases providing an end-to-end solution is simply too much, especially as a starting point. But catering to the lowest common denominator, for example, providing only the enabler technology (which seem to be the strategy you've taken), leaves very limited choices in the hands of the customers. This in my view is another challenge of scalability, only this time I mean scaling of value proposition and cost :. The GigaSpaces approach has been to provide three editions: 1. Free Community Edition: provides the API implementation (JMS, JDBC, JCache, JavaSpaces) without full clustering capabilities. With this edition users can take advantage of our implementation, build their application, and even move to production, totally for free, and knowing that at any given time you can scale up to another edition just by changing your license key. I was actually surprised to see in JavaOne that many users have taken this approach. 2. Caching edition: probably the equivalent of what you mentioned, i.e. an embedded solution that can be embedded with any application, be it J2EE, Grid, etc. As it stands today I would say that 50% of our users have taken that approach, starting from pure OEM deals (some with the biggest software vendors running hundreds of installations, without anyone knowing that it's the software is running on top of GigaSpaces) to large organizations that are using it to run mission-critical applications. 3. Enterprise Edition: provides the end-to-end solution with built-in SLA and the ability to manage middleware together with the business logic associated with it. What's nice about this approach is that you can switch between them ("scale up":) just by switching a license key. We already have customers that started with the Caching Edition and switched to the Enterprise Edition. Virgin Mobile would be one of them. Recently we started to see that this happens even during the evaluation process; users start evaluating the Caching Edition, and as soon as they are exposed to the capabilities of the Enterprise Edition they decide to switch to it. To be clear on what we're not - we are not trying to be a general purpose grid vendor or even an enterprise grid vendor, and the same goes for J2EE. We focused on building a grid-aware middleware with built-in SLA capabilities, which solves the end-to-end scalability in the application layer by solving both processing and data access bottlenecks. We partner, work with and complement other grid or application server providers, and many of our customers use GigaSpaces alongside Platform or Data Synapse. We've also made our middleware available through the Sun-Grid center. Nati Shalom CTO GigaSpaces Write Once Scale Anywhere
  10. Scalable Performance[ Go to top ]

    Hi Nati - For the meat of your argument ;-)
    In my view [..] you cannot guarantee availability without providing proper deployment and management capabilities. You become an enabler, albeit an important one, but just an enabler that leaves much of the complexity in the hands of the application developer.
    Our customers have hundreds of different infrastructure solutions in place for starting up and monitoring processes (e.g. JVMs). They are not asking for yet another stand-alone GUI for managing yet another silo of servers. What they want is for a solution to fit into their existing infrastructure in as seamless yet as serviceable and manageable a manner as is possible. That's what we do. Deployment and management are usually in the realm of "operations", not development. In that regard, Coherence was designed explicitly for "Lights Out Management / Zero Admin" (LOM/ZA) environments. It doesn't require any additional hardware, processes, GUIs, consoles, and doesn't even need an "installer" ;-) .. it is just placed inside the application by the developer, and at runtime it transparently and automatically joins into the peer-to-peer cluster of application instances running within the grid. (And it's been that way since 2001.)
    The end result is that you cannot maximize potential performance and scaling gain of the available resources.
    As you know, we have supported dynamic capacity of data management since our Coherence 1.2 Clustered Edition release back in June of 2002. As you add servers, we automatically load-balance state management (including transactional load) across those servers, without interruption and without data loss. From a quick read of your support forums, this is something that isn't even available in your 5.0 "enterprise version" release. At some point in the future, your enterprise version may be able to provide the basic capabilities that we had in 2002, and then we'll have a common basis to talk about scaling to use available resources.
    Having said that, I do agree that in some cases providing an end-to-end solution is simply too much, especially as a starting point. But catering to the lowest common denominator, for example, providing only the enabler technology (which seem to be the strategy you've taken), leaves very limited choices in the hands of the customers.
    Wow, so many things .. I don't know where to begin ;-) First, we have always believed it to be far more important to provide a working solution that addresses the real problems that our customers are facing, than to propagate the notion of an "end-to-end" solution. We would rather do one thing right than a whole bunch of things half-baked. The market seems to agree, with Coherence continuing to add more customers each quarter than the other mentioned products in this thread have garnered in total. Second, I find it humorous that you use the term "least common denominator", since that term is often reserved for products that attempt to be all things to all people. It is unwise (from a debating perspective) to accuse us of "focus" and "least common denominator" in the same paragraph .. try spreading those two arguments out a bit more ;-) Lastly, customer choices are expanded (not limited) when the products they are using are flexible enough to fit into their own unique environments. Obviously, you are passionate about your positions. At the end of the day, I have no desire to dissuade you from your beliefs and from your strategy. However, it is quite disingenuous to be so critical of our strategy, when it is clearly working so well. Yes, it's true that as a result of our focus on data grids, in-memory data management and caching, we will no doubt miss some opportunities in the market. If you can address those opportunities, then I wish you all success with them. Peace, Cameron Purdy Tangosol Coherence: The Java Data Grid
  11. Re: Scalable Performance[ Go to top ]

    Cameron - I have learned to respect you over the years and I am genuinely pleased that both of our companies continue to prosper. At this point, I encourage the community to try out all leading solutions and keep an open mind: multiple options have a valid place in the markeplace. Nati Shalom CTO GigaSpaces Write Once Scale Anywhere
  12. He mentions Tangosol briefly for just its clustered caching features, but check out their grid features: http://wiki.tangosol.com/display/COH31UG/Scaling+Out+Your+Data+Grid+Aggregations+Linearly
  13. Getting the data ready[ Go to top ]

    Nice. The issue though is getting the data into the grid, viz - loading the Trade objects into memory. The raw data to be processed by most heavy-duty batch tasks are buried into a relational database like a datawarehouse. A key requirement to be able to leverage a data grid is that your data fits nicely and naturally into convenient domain objects ("Trade" in this example) - hardly an easy task when dealing with legacy data warehouses. I wish this were possible, but alas, we are doomed to running a bunch of heavy-duty SQL and stored procedures.
  14. Re: Getting the data ready[ Go to top ]

    The issue though is getting the data into the grid, viz - loading the Trade objects into memory. The raw data to be processed by most heavy-duty batch tasks are buried into a relational database like a datawarehouse. A key requirement to be able to leverage a data grid is that your data fits nicely and naturally into convenient domain objects ("Trade" in this example) - hardly an easy task when dealing with legacy data warehouses. I wish this were possible, but alas, we are doomed to running a bunch of heavy-duty SQL and stored procedures.
    Legacy problems will always be legacy problems ;-) In the "trades" case, the trade information related to what's going on today is already in the data grid, because that's where it was created .. it only gets into the data warehouse because the data grid puts it there. Unfortunately, that's the difference between building a new system versus trying to grab data out of an older one. Peace, Cameron Purdy Tangosol Coherence: The Java Data Grid
  15. Re: Getting the data ready[ Go to top ]

    because that's where it was created .. it only gets into the data warehouse because the data grid puts it there.
    Sorry to post so late to this thread, but .. So, is there any reason for a data warehouse any more? And, any white papers on this subject (ie - the Data warehouse being the receipient not the source).
  16. A few comments . . .[ Go to top ]

    Tim Bray makes some excellent points in this article. I particularly like the descriptions of the advantages of memory-based operational data storage and where disks still shine (anything involving sequential data-writes). I have a couple of thoughts and comments to add as grid architecture and use cases are evolving very rapidly right now. - I am surprised you didn’t find any data grid products to meet your needs. The newest releases are packed with grid-targeted features derived from large-scale grid planning and deployment experience. These types of features include o Higly-tunable options for data distribution (sync, async, Txn’s with ACID characteristics, durable store-and-forward, tcp/multicast/unicast, etc) o Flexible and dynamically adaptable data allocation strategies, including massively scalable partitioning of huge data sets and ways to move tasks to data rather than the other way around. o Support for multiple languages o Highly tunable concurrency/contention optimization o Highly tunable network resource optimization, o Low-latency continuous querying, optimized disk persistence (as described in the article, and more) o And the list goes on . . . - The “Predict Financial Risks and Returns” example began with batch but is rapidly evolving to real-time or near real-time with the availability of mature data grid (or Data Fabric) technology. Projects are in planning or have begun at many global banks driven by regulatory and competitive pressures and include hybrids of the landscape categories described. In your summary, you state that “the right way to talk to it is to get a hostname/port-number combination, call it up, send a request, and wait for a response.” The problem with this is that even this overhead is a serious performance drain that doesn’t meet the performance requirements or SLA’s of many applications. Where your service access occurs, a connection should already be available, and smart configuration can allow you to service grid tasks with minimal network hops and artificial bottlenecks—often as the result of pushed data with no request/response intermediation. A little further down, you state that “I don’t want too many layers of abstraction between me and the messages. My feeling is that server failures are going to have to be handled in an application-specific way, not pushed down into the abstraction layer.” I agree with wholeheartedly that every application has specific and different requirements, but solutions are available where you can maintain strict logic partitioning and robust HA/Failover without a performance penalty. At least in finance, the forward thinking trend is towards grid infrastructure dynamically supporting large numbers of different applications on a utility model, thus maximizing the scope of use-cases with the most flexible infrastructure management capability. Significantly different applications will need to run on these grids in parallel and at different times of day. Some may be SOA deployments, some are huge Value-at-Risk jobs, some are Portfolio Valuation models, and some involve real-time support for complex derivates trading. The limitation so far, however, is that all of the examples above suffer from data starvation when scaled-up on grids—performance gains from additional hardware tend to flatline fairly quickly. Dynamic and complex applications are especially challenging without a robust data grid if they have high demand for ephemeral data storage and distribution and/or contentious and highly parallelized transactions. The commercial grid schedulers have gone far towards solving the task parallelization and dispatching problems (and much more), but look for grid capabilities and massive scale for a much broader basket of use-cases to really take off over the next couple of years—aided by tight robust data grid integrations and several emerging and synergistic network technologies. This will also clearly goes far towards validating the GridBus vision of a powerful economic model for grid utility-Utilization will eventually be maximized for $$$ reasons. Lots of great info here, but I think that there is a fundamental limitation in the type of hands-on evaluation one can achieve without access to a high number of servers and a team of specialists to support it. I’ve seen many examples where design/architecture decisions that were based on limited-scale tests (10’s, rather than 100’s) must be reversed or considerably altered—largely because of data grid issues. Cheers, Gideon Low GemFire-The Enterprise Data Fabric http://www.gemstone.com/downloads
  17. Discussion Thread hijacked?[ Go to top ]

    It would seem we may want to rename this tread to "Battle of the Data Fabric vendor CTO's". True to the fabric metaphor, there seems to be a fair amount of stretching going on [name withheld], but in this case due a rabid outbreak of marketing-speak. We should all promise not to bullet-list our company products' features and editions in these threads to the benefit of the entire TSS community. Cheers, Gideon Low GemFire--The Enterprise Data Fabric www.gemstome.com