Making 'big data' projects flexible, timely with Agile software development

Agile can help in the delivery of more timely applications in 'big data' projects, largely due to better collaboration between business and development.

Agile's capacity for handling rapid change can help bring real-time, high-volume reporting to projects involving "big data," said Ian Abramson, director of enterprise data at EPAM Systems, based in Toronto. At EPAM, Agile usage results in the delivery of more usable and timely applications, largely due to better collaboration between business, development and data analytics leaders.

Why are Agile software development and big data a good match? At Oracle OpenWorld last week, Abramson laid out the reasons why Agile is a good means to besting big data challenges.  

"Agile centers on responding to change over following a plan, on what's the most important thing to you right now," said Abramson. "Agile practices can help developers bring real-time predictions, insight-driven optimization and unified views to big data reporting tools."

Big data analytics and management tools are commonly used today for business strategy planning, inside user-facing applications such CRM and ERP, and in support systems such as data warehouses and recommendation systems.

Today, big data tools are bringing together information, both structured (as in SAP) and unstructured (think Twitter), to augment business leaders' knowledge about the company, their customers and how to do business well. Big data helps businesses understand how they communicate and consume in order to drive actions and choices in the future, said Abramson.

Usage of data analytics and management tools is widespread in operational and interactive applications today, but the two groups are often siloed. It's time to join the two and create game-changing approaches to data management, business analytics and application development, said Abramson.

Leadership and collaboration are the biggest hurdles to integration and innovation in big data technologies. "We need business and technology leaders who understand what big data is about and can work together," said Abramson.

Businesses need reports that effect change and are actionable to reach the golden mean, business value. High quality and effectiveness are hard to achieve in data analytics, largely because information and its sources are varied and often uncertified. "A single version of the truth is difficult to achieve," said Abramson. To get closer to that truth, business intelligence (BI) and data analytics have to be integrated with business processes. 

Abramson has used Agile practices successfully to bring together EPAM technologists and business leaders. "Collaboration among functional users, business strategists, development and IT provide a conduit to high data and product quality," he said.

Using Agile's diverse team-building practices -- such as daily stand-up meetings, collaboration tools and the creation of a common language -- helps build trust within the team. Agile also helps mitigate project risks, since it requires constant iteration planning, iteration reviews and user feedback. Reducing risk is a critical trust-builder for businesses because data integrity is critical to success.

Agile works with big data because Agile is iterative, incremental and evolutionary, Abramson said. Working software is delivered on a regular basis. Continual improvement is enabled by short iterations and constant user feedback.

The revolutionary social media landscape is constantly changing how data is gathered and consumed. A key focus of Agile development is working on what matters to the user first, so the stage is set for the rapid development of new types of reports to handle changes to data technologies.

"To effectively use big data today, business has to change on a dime," said Abramson. "With Waterfall, by the time you finish building a report, you're behind."

Abramson offered a comparison of deliverable timelines for traditional, Waterfall and Agile data projects based on his own experience. Waterfall development delivered five extractions, transformation and loading (ETL) entities a week and were released quarterly, compared with Agile's 10 entities and biweekly releases. Using Agile allowed the team to increase deployment of features 25% more quickly than could be done with Waterfall.

Pairing Agile and big data projects promotes understanding of how data knowledge benefits the business, said Abramson.

More on big data

Storage, network and server challenges with big data analysis in the cloud

'Big data' among top annoying data management buzzwords

Challenges of big data for traditional analytics approaches

Dig Deeper on Software programming languages

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.