How AI can help with requirements analysis tools
Functional requirements and user stories create the blueprint for software applications. A good set of requirements can simplify developer’s work when they translate business ideas into working codes. Teams spend a lot of time gathering and organizing user stories into requirements that guide developers in writing code, but often important details get left out and important functions are duplicated in slightly different ways.
However, AI and natural language processing (NLP) are now more commonly used to identify problems in user stories and requirements as they are written. These technological advancements provide clearer feedback to the business, and in turn send better user stories and requirements to the development team. It can also help improve to plan software development schedules.
IBM’s Engineering Requirements Quality Assistant uses Watson for requirements analysis managed on the DOORS Next platform. ScopeMaster has developed a stand-alone requirements analysis tool that plugs into Jira, Azure DevOps and Pivotal Tracker. These kinds of tools promise to complement the AI usage to improve other aspects of application development tooling.
Lonnie Franks is an independent consultant that’s managed larger software projects since 1978. He said that good requirements is one of the most important aspects of successful software development programs. Franks finds that enterprises typically spend an extra 60% dealing with requirements problems on larger projects because of requirement defects. Even with smaller Agile development cycles, bad requirements create various types of problems to address.
Franks has worked with ScopeMaster to help analyze requirements and believes that AI requirements analysis will play a major role on how to improve both small and large software development projects.
The root cause of failure
Franks discovered how important requirement quality while managing software development for NCR — an omni-channel provider — from 1978 to 2000, where he never had a failed project. When Franks started he was an expert at software development but hadn’t yet mastered software project management. After his predecessor was fired, he was promoted to engineering manager.
“Knowing that I was anything but qualified, I worked hard to learn how not to get fired,” Franks said. He discovered that most projects that grew beyond $2 million had a greater chance of failure than success. One of his key discoveries was how important good estimation was up front. In the larger projects, rework ended up about 60% of the total cost.
Franks’ first big project was with a large bank in Cincinnati and he told the CEO they needed to redo the requirements. The CEO was upset, but after Frank’s team took the time to redo the requirements, they delivered the project on time and on budget. His team only discovered three defects in the first six months in over 2 million lines of code. “I knew that getting requirements right and understanding the size was critically important,” he said.
Augment software analysis
When Franks first heard about using AI for requirements analysis, he experimented with ScopeMaster to size a project and identify problems. He quickly found 150 defects and was able to generate a clean set of requirements in a couple days.
“I was impressed, it would have taken me weeks to do that manually,” Franks said. “Not only did I get a function point count, but it helped me to figure out how many programmers, business analysts and testers should work on the project.”
One area where this kind of AI-driven analysis excels is missing requirement identification. Franks has found that projects with about 1,000 function points should cost about a $1 million and take a year to execute and require about 50,000 lines of code in C# or Java.
However, these projects will also have about 1000 requirements defects.
“That seems impossible, but since I have been managing software engineering for multimillion projects for about 40 years, that ends up being about right,” he said. A great deal of these defects are missing requirements to implement the complete set of Create, Read, Update, Delete operations in a program.
Franks found that he can use ScopeMaster to find and fix about 500 defects in a week. “If you can fix half your requirements in a week, you can suddenly be on time and on budget with 100% of your requirements done,” he said.
Different defects for Agile
Agile development teams tend to be more efficient than their waterfall development counterparts and are generally able to develop higher quality software. In Franks’ experience Agile teams have about .6 defects per function point. The good thing is feedback comes a lot faster, but doing Agile projects also can result in a lot of rework as a result of requirement defects. “Most of the rework has to do with ambiguous requirements rather than missing requirements,” he said.
These problems are compounded for scaled Agile teams where coordination between multiple teams is hard. Teams can centralize requirements to help coordinate them since Agile and Scrum teams don’t tend to generate as much shared documentation.
Franks recently worked on a slot machine application development project that was spread across multiple teams. He discovered they had implemented three different fields to keep information about the jackpots implemented by three different teams. Once he identified this, he had them use the same name and helped cut work down by a third. “Once you use terms consistently, you don’t have duplicate work. With agile sometimes you do duplicate work because you don’t know what terms may be inconsistent,” he said.
Types of defects in user stories
The NLP engines in these requirement analysis tools evaluate the text of user stories and then cross references the results with all the other user stories in a set. It detects the functional intent from each user story.
Colin Hammond, CEO of ScopeMaster said they see 10 types of defects in functional user stories. Today AI can detect defects related to ambiguity, inconsistency, complexity, duplication, omissions, testability and sizeability. However, AI struggles to find defects such as whether a requirement is valid, design laden or current.
The frequency and impact of requirements defects by type varies. Hammond said they typically see seven times more omissions than duplicates. Omissions are what cause projects to go over the original estimate, budget and timeline. Ambiguities are particularly harmful, and often mask other defects. Hammond recommends that teams focus first on disambiguating requirements. Then it’s easier to remove any subsequently exposed defects.
Still in its infancy
Using AI to analyze and process requirements and user stories is still in its infancy. Franks said none of the tools have domain knowledge. For example, when he works on an ATM project for banks, these tools aren’t able to catch things that are specific to how ATMs work.
Also, they’re not able to translate user stories into code. While user stories are the foundation for requirements, developers must also understand nonfunctional requirements and UI issues. Today, AI requirements analysis tools excel in guiding developers to set up packages, classes and methods from the user stories. However, other kinds of platform-specific code generators could help fill in the gaps for things like UI element generation on the front end.