Maximizing and managing big data with SOA middleware
A comprehensive collection of articles, videos and more, hand-picked by our editors
There are huge amounts of information available to application developers today. However, the act of gathering big data has limited usefulness if it's difficult to access, organize, transmit, decipher, analyze or put to use. Managing big data is critical, and staying on top of the latest big data trends keeps application developers in control.
By submitting your personal information, you agree that TechTarget and its partners may contact you regarding relevant content, products and special offers.
In 2012, many enterprises woke up to the fact that their data is more than a collection of information to be tucked away in a database -- it is an incredible resource. In 2013, we anticipate that big business will start to streamline and mine big data to unleash its hidden value. But the accumulation of even more data presents increased costs and new challenges for application developers. Here are the three major big data trends for the coming year.
Wouldn't it be nice if all your data could be stored in neat little boxes in a relational database?
Where do we put all this stuff?
According to IBM, 90% of the data currently stored in data centers across the globe was created in the last 2 years. By 2020, the total amount of data may increase 50 times over. Then, computers will become sentient and take over the world with the goal of exterminating the human race. In the meantime, enterprises are going to have to figure out how to scale out and find room for all that data.
There's only so much an organization can do to improve storage utilization and management on its own. Virtualization has helped a lot in the past. Even so, IT is going to start running out of floor space/rack space and running up the electrical bill on a whole new level if they are tasked with storing big data in-house.
Cloud storage using Infrastructure as a Service/Platform as a Service is the obvious solution, and one that even late adopters will have to consider. As big data grows even bigger, storing all of their own data will no longer be an affordable option for the enterprise. In 2013, businesses may find that the cloud isn't just the most cost-effective solution -- it may be the only feasible way to store the amount of data coming in.
Major cloud providers that are aggressively annexing and building data storage space (and figuring out even more novel ways to save a kilowatt) will seem more and more attractive as enterprises look for new places to store their data overflow.
Unstructured data will reshape big data
Wouldn't it be nice if all your data could be stored in neat little boxes in a relational database? "There's a green one, and a pink one, and a blue one, and a yellow one, and they all fit in MySQL and they all look just the same." Unfortunately, the huge uptick in data that enterprises gather in 2013 will have a distinctly unstructured look and feel. It's not just documents -- we're looking at image files, audio, video, social media actions and ever-growing amounts and varieties of embedded sensor data. A relational database is obviously not going to cut it.
There's a green one, and a pink one, and a blue one, and a yellow one, and they all fit right in MySQL and they all look just the same.
Harish Ramachandran, co-founder and project manager at Santa Clara, Calif.-based CIGNEX Datamatics, pointed out that big businesses have painted themselves into a corner by failing to predict the future (shame on them!). "Enterprises are confronted with this challenge of the volume of data and the variety of data and the sort of use cases that their back-end systems and content repositories were not originally architected for."
The solution is not going to be simple or singular. Instead, enterprises are moving toward the development of a database ecosystem. It's not just a question of public cloud, private cloud, or on-premises storage. Different platforms and management tools are going to have to be integrated to create a comprehensive solution for multiple data streams. This is likely to include a hybridization of SQL and NoSQL solutions.
Businesses that want to grow with their data will have to move beyond the concept of data as something that exists in a warehouse and start thinking in terms of data as a process or a living organism. If recent history speaks to the future, content management (including the portal or integration layer) will be just as important as data storage.
Big data is only as good as your analytics
Speaking of predicting the future, that's actually what big data should be doing for businesses. Analytics will really start to come into its own in 2013 in both real-time and predictive forms. Some companies are already doing this -- and other businesses should pay attention.
Paul Jones, business systems architect at dental insurance provider United Concordia, pointed out that enterprises can use analytics to leverage big data in many different ways within the space of a single interaction."Some companies are using predictive analytics in customer service to determine the next, best action for a call. Or, they can determine how best to market the company to the customer or how to resolve their issue."
He also noted that analytics gets smarter when you have access to more information (even bigger data). "For the first time, we're integrating third-party information into our current information to get a better picture, a 360-degree view of the customer, in order to better serve and understand them and in order to evolve our business."
Follow us on Twitter (@TSS_dotcom)
The future of big data analytics will depend on its ability to scale and diversify to sort out the signal from the static in an ever-increasing field of data. And it's going to have to do this mammoth task faster and faster with fewer and fewer resources. In fact, we expect to see mobile devices that incorporate more and more on-board capabilities to bring real-time analytics out of the enterprise and into the real world very soon.