|
Sponsored Links
Resources
Enterprise Java
Research Library
Get Java white papers, product information, case studies and webcasts
|
News
News
News
|
Messages: 7
Messages: 7
Messages: 7
Printer friendly
Printer friendly
Printer friendly
Post reply
Post reply
Post reply
XML
XML
XML
|
 |
20 Real-Life Challenges in Cloud Computing
Nikita Ivanov from GridGain Systems, an Open Source Grid Computing Middleware, in his recent blog has shared his experiences based on some extensive work on Cloud Computing at GridGain (i.e. deploying Java grid application on cloud environment), and described a distinct set of problems and challenges associated with that work whether it is related to the cloud computing in general or to grid middleware you are using on it.
Here is a short list in no particular order that he had accumulated over a year by now and something that drives many of the improvements in GridGain that are currently in the works:
1. Most likely you do NOT need cloud computing – but if you do you would know it for sure by now; people who have legitimate both technical and business-wise use cases for cloud computing have been trying to do it internally for many years
2. The best way to think about cloud computing is “Data Center with API” – that should clarify most of the questions…
3. Creating the image for something like Amazon EC2 is worth about 45 minutes of your effort – but you will spend weeks and months after that fine tuning your application and developing additional functionality; plan accordingly
4. You are about to deal with 100s and 1000s of remote nodes… Things that worked in 10s of nodes often “mysteriously” don’t work on the “cloud” scale. We were surprised by the amount of configuration tweaks we had to make to run GridGain on 512 nodes on Amazon EC2 under the load. Proven grid middleware is essential (quite obviously)
5. You cannot rely on the fact that environment will be homogeneous – most likely it will be not: different CPUs, different amount of memory, etc.
6. Debugging problem on that scale requires pretty deep understanding of distributed computing; learning curve is very steep; trial and error is often the only solution; plan accordingly
7. IP multicast will likely not work or work with significant networking limitations. For example, you may not get all the computers in your cloud in the same IP multicast group. QoS on IP multicast can be unknown, at best
8. Traffic inside is very cheap or free – but traffic outside is expensive and can “get you” very quickly
9. If you have to use cloud all the time – economics go down and in many case it is cheaper to traditionally rent in data center; that means that in many cases using clouds is best as an options to “outsource” pick loads – in such cases the economic effect can be dramatic
10. Up time and per-computer reliability is low – comprehensive failover support on grid middleware is a must
11. Static IPs are not guaranteed – it kills automatic deployment for 90% of the grid framework out there
12. Almost always plan on having multiple clouds, at least one internal and one or many external; you are always going to have data and processing that cannot cross the boundaries of your internal data center; without comprehensive support from grid middleware for location transparency (a.k.a. virtual cloud) this is a show stopper
13. External clouds (i.e. hosted NOT by you) present problem of sharing data:
- Do you copy data to the cloud?
- Usually no local-DB access from the cloud
- Can you legally copy the data?
- Double storage of data locally and on the cloud?
- Synchronization?
- Security?
- Data affinity?
- Local data is removed once image is undeployed…
- Etc, etc.
14. Carefully think through dev/qa/prod layout and how this is all organized – things get way hairy with multiple cloud, etc.
15. Clunky (re)deployment of your application onto the cloud can slow down development process to a halt – support from grid middleware is absolutely essential here
16. Often connections are one-directional, i.e. you can connect to the cloud – but NOT from the cloud back to you – comprehensive communication capabilities supporting one-directional connectivity and disjoint clouds in grid middleware is a must
17. Cloud are implemented based on hardware virtualization – make sure your grid middleware can dynamically provision such images on demand, i.e. basically start the image (start paying) when certain conditions are met and stop the image (stop paying) when other conditions in your system are met
18. Stick with open source stack (no, this is not a plug) – having a source code helps greatly during debugging in such unusual situations
19. Linear scalability can only be achieved in a control test environment (like in our recent test) – real world applications will exhibit some sort of non-linier scalability; it is essential to have at least a ballpark number of what you are expecting the scalability and performance should be when you run your application on the cloud – battery of performance and scalability tests developed upfront is usually the best option
20. Personal recommendation: use Amazon EC2/S3 services – the best offering at this point by a long, long mile
|
|
|
Message #268085
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: 20 Real-Life Challenges in Cloud Computing
Most likely you do NOT need cloud computing – but if you do you would know it for sure by now; people who have legitimate both technical and business-wise use cases for cloud computing have been trying to do it internally for many years
That's a ridiculous statement. I think almost anyone, other than someone who can run their dev, integration and prod environments on one box, can somehow benefit from cloud computing. When referring to on demand scalability feature, then maybe only apps that have variable scalability requirements can benefit from them.
1. You can benefit from clouds automatic provisioning and management of nodes. Say you have a QA cycle that requires more instances than usual. Yes, you can buy the hardware, but why? Just provision a node within 2 minutes on EC2 and QA now has a completely isolated box to do their testing on. Ah, and you can shut this box down 10 hours later, owing only a few dollars and never have to use it again.
2. Yes, setting up clouds is not a matter of starting a node, it takes hours or days to then configure it, install all necessary packages, create some automated scripts, etc... But this work is a benefit for lots of money saved by provisioning on demand hardware that you now won't have to maintain in house.
3. Running an application on a cloud is not as straight forward as just starting multiple nodes. The application or its middleware has to be cloud aware. For the most part it means that it has to be horizontally scalable without any reliance on a static node configurations. If your application is capable of doing it now, it can most likely run in a cloud environment. Nikita has a great point though, running an app on a few nodes is different than running it on 100 nodes. But most likely you won't need 100 nodes and if you do, you probably don't want to write the supporting infrastructure yourself. There are many middleware products out there now that are cluster away. GridGain, GigaSpaces (my favorite), and many more...
Ilya
|
|
|
Message #268096
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Not that simple...
That's a ridiculous statement. I think almost anyone, other than someone who can run their dev, integration and prod environments on one box, can somehow benefit from cloud computing. When referring to on demand scalability feature, then maybe only apps that have variable scalability requirements can benefit from them.
You can benefit from clouds automatic provisioning and management of nodes. Say you have a QA cycle that requires more instances than usual. Yes, you can buy the hardware, but why? Just provision a node within 2 minutes on EC2 and QA now has a completely isolated box to do their testing on. Ah, and you can shut this box down 10 hours later, owing only a few dollars and never have to use it again.
That is usual over-simplification in my opinion. If you would work with deploying on clouds you would see plenty of actual challenges this break this ideal picture (many of them mentioned in the original post, by the way).
I'm very optimistic, however, about overall promise and technology behind cloud computing (i.e. deploying grid applications on cloud infrastructure) - but I think the middleware stack has to come a long way to make it a much simpler reality. We at GridGain (can't speak for others) are working very hard on pretty much a break-through technology & approach to this problem and we'll be releasing it in the next 3-5 months.
In a nutshell, we will bring the same simplicity that GridGain users are enjoying right now to a seamless and on-demand running (scaling out) of your grid applications on the cloud infrastructure. What's even more unique about is that you will be able to do all that without ever leaving your Java IDE or emacs or vi - the same straightforward approach and unique capabilities like grid P2P class loading, location transparency, adaptive load balancing, etc, etc. will still be available out-of-the-box.
Stay tuned,
Nikita Ivanov.
GridGain - Grid Computing Made Simple
|
|
|
Message #268097
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: 20 Real-Life Challenges in Cloud Computing
Most likely you do NOT need cloud computing – but if you do you would know it for sure by now; people who have legitimate both technical and business-wise use cases for cloud computing have been trying to do it internally for many years
That's a ridiculous statement. I think almost anyone, other than someone who can run their dev, integration and prod environments on one box, can somehow benefit from cloud computing. When referring to on demand scalability feature, then maybe only apps that have variable scalability requirements can benefit from them.
Hi Ilya,
I you think about it, what is even more ridiculous is arguing over a term as ill defined as "Cloud Computing". It seems to me that this term can mean almost anything you want it to mean :)
It is this loose language that gets us into trouble and allows room for marketing speak void of real technical content, which is probably what this article is really about.
Why not be more specific about what you mean? I like the term Software-as-a-service, which doesn't need Amazon EC2 or GridGain incidently:) Or even better code-on-demand from Roy Fieldings REST Dissertation. Loose talk, loose minds...
Food for thought?
Paul.
|
|
|
Message #268100
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: 20 Real-Life Challenges in Cloud Computing
It is this loose language that gets us into trouble and allows room for marketing speak void of real technical content, which is probably what this article is really about.
I am failing to see a marketing pitch here. The post clearly shares the experiences an individual had when deployed a large grid infrastructure onto Amazon EC2 cloud.
Would you rather prefer if the post read as follows: Nikita Ivanov from unknown company with unknown experience in Cloud Computing has vetted in his blog about cloud computing and is sharing with us experiences we are not sure he ever had?
Lighten up. This post is definitely not an PHD thesis on cloud computing, but surely does provide useful pointers about the subject.
Best,
Dmitriy Setrakyan
GridGain - Grid Computing Made Simple
|
|
|
Message #268118
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: 20 Real-Life Challenges in Cloud Computing
It is this loose language that gets us into trouble and allows room for marketing speak void of real technical content, which is probably what this article is really about.
I am failing to see a marketing pitch here. The post clearly shares the experiences an individual had when deployed a large grid infrastructure onto Amazon EC2 cloud
OK. So why not call it: "An Article on my Experiences Deploying a large Grid Infrastructure onto Amazon EC2"? This would be so much more useful, especially if he gave some more context about precisely the type of Application upon which his experience was based.
Of course I have no means of knowing the motives of the author, and beyond my speculation they could be very noble. My real point is about language and the descriptive power of the term "Cloud Computing".
If you speak to psychologists they will tell you that language is a very important part of the thought process. In fact without language thought itself may be impossible, yet in Software we use loose, imprecise language all the time. Is it any surprise that people end up talking (arguing) past each other?
Paul.
|
|
|
Message #268151
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: Not that simple...
I do work and deploy on clouds and have faced many challenges on getting an application to be "cloud compliant", if there is such a thing. If nothing else, the lack of good SLAs in return for being able to dynamically bring up nodes without any static mappings is a challenge to any traditional application that relies on such facilities. Relational dbs for example are currently not cloud compliant. Yo;u can easily deploy one on a single node, but if you want to do any sort of replication, it requires a more static infrastructure with greater availability and it can't reprovision itself if failure/reconfiguration of the cluster occurs in a cloud at runtime.
But what I see you constantly push is grids, of course I understand your bias. Grids are great, I recently wrote an app based on GigaSpaces (they work flawlessly on ec2). Not everyone needs grids and there are many more use cases for clouds than grid computing.
Ilya
|
|
|
Message #268152
Post reply
Post reply
Post reply
Go to top
Go to top
Go to top
|
|
Re: 20 Real-Life Challenges in Cloud Computing
Paul, completely agree. Cloud computing, which amazon is trying to make synonymous with ec2 is a loose term. Maybe I'm really speaking of ec2, which is much more than just resources in a virtual cloud. I think their resource provisioning and management facilities can benefit any datacenter, whether in production or just in qa/devel phase. So I agree, I should change everything in my reply to rather say "ec2".
|
|
|
| New content on TheServerSide.comNew content on TheServerSide.comNew content on TheServerSide.com |
|
|
Reza Rahman continues to explore the features of the proposed JSR 299, Contexts and Dependency Injection for Java EE (CDI). When approved, it promises to be a key feature of Java EE 6.
(January 21, Article)
Ted Neward is an independent consultant specializing in high-scale enterprise systems, and an authority in Java and .NET technologies. He is the author and co-author of several books, including Effective Enterprise Java. At TheServerSide Java Symposium in March, he will be presenting sessions on pragmatic architecture, ECMAScript and Scala.
(January 15, Article)
Now that Oracle is absorbing Sun Microsystems, there mixed views on what should come of the Java Community Process (JCP). While some say Oracle should become the new steward of Java and keep the JCP much as it was, others argue that it may be time to open-source this widespread language.
(November 24, Article)
Reza Rahman explores the features of the proposed JSR 299, Contexts and Dependency Injection for Java EE (CDI). When approved, it promises to be a key feature of Java EE 6.
(November 2, Article)
SAML is an XML-based standard for exchanging authentication and authorization data between security domains. The single most important problem that SAML was created to solve is the Web browser Single Sign-On problem. Many organizations are debating whether to stay with version 1.1 or move to 2.0. This article makes observations about both options.
(September 28, Article)
Joe Ottinger takes a look at how people learn, and applies it to the practice of programming. He notes that understanding how people learn is an essential part of working in a programming team.
(September 22, Article)
Stephen Maryka gave us an article about the Asynchronous Web and posed a number of questions that get examined like an approach to delivering Asynchronous Web capabilities through extensions to existing Java EE technologies.
(July 14, Article)
JavaServer Faces Flex goal is to provide users capability in creating standard Flex components, part of flexSDK which is open sourced through MPL license, as normal JSF components. This article by Ji Hoon Kim will provide an overview of creating a simple multilingual JSF page consisting of JSF Flex tags.
(June 29, Article)
In this session Jeff explores the key characteristics of successful SOA projects. He covers some of the patterns, and anti-patterns, tool sets, and strategies that he himself learned the hard way. Last, he provides a strategy and blueprint for achieving a high likelihood of success in your SOA project.
(June 23, Tech Talk)
Ari Zilka, CTO of Terracotta, Inc., talks about the new features in Terracotta 3.1, announced during JavaOne and available now.
(June 15, Tech Talk)
In this Tech Talk, Josh Long explores an integration challenge using Spring Integration and walks through the implementation, employing and expanding on the basic patterns of Enterprise Application Integration to tie together components into a function integration solution, and then demonstrates how Spring Integration helps address the integration requirements.
(June 15, Tech Talk)
In this Tech Talk, David Geary teaches you: The basics of Google Web Toolkit; How to implement Ajax-enabled applications in Java; Internationalization; Hooking into the browser history mechanism; Remote procedure calls.
(June 4, Tech Talk)
Jon Kern discusses the best architecture/technical solutions and ensure that they are repeated by all developers. By tackling the architecture up-front in a serial manner, subsequent parallel development will be much more manageable and predictable.
(May 28, Tech Talk)
This keynote describes the frustrations of modern knowledge workers in their quest to actually get some work done, and solutions for how to guard yourself against all those distractions. Neal Ford talks about environments, coding, acceleration, automation, and avoiding repetition as ways to defeat the misguided attempts to sap your ability to produce good work.
(May 26, Tech Talk)
Gil demonstrates how new, aggressive uses of already abundant compute capacity by common applications offer competitive value for application designers.
(May 21, Tech Talk)
Chris Keene introduces WaveMaker as a new way to automate the ability to generate Hibernate classes in order to more quickly bring OR mapping into an application.
(May 19, Article)
Download the entire book of Jakarta-Struts Live and learn about Struts MVC, Tiles, the Validator, DynaActionForms, plug-ins, internationalization, and more.
(Book PDF Download)
The Application Server Matrix is a detailed listing of J2EE vendors and their application server products, with information on latest version numbers, J2EE spec support and licensing, pricing, platform support, and links to product downloads and reviews.
(Application Server Comparison Matrix)
|
|
