Tierney - stock.adobe.com

How to scale automation in DevOps environments

With the right tools and processes, programmers can automate at scale many of today's manual tasks. Here's how one company developed automation-at-scale best practices.

The telecom giant CenturyLink created a federated program to scale automation efforts, where one dedicated team curated standards and best practices that were then rolled out across the company.

CenturyLink -- a facilities-based communications company with 40,000 full-time employees, based out of Monroe, La. -- wanted to achieve automation at scale and blend the best characteristics of application development, scripting, and robotic process automation (RPA) tooling, said Troy Ferrence, senior manager of automation solutions architecture at CenturyLink.

For the past several years, Ferrence's team has focused on scripting and application development, but its efforts accelerated when CenturyLink introduced RPA software that used the UiPath platform. CenturyLink expanded its work to more than 19 teams dedicated to this project, referred to as center of excellence teams, in the last year and developed over a thousand different automations.

 "We have a lot of legacy applications where we don't have any type of interface or availability for enhancement which are perfect candidates for RPA," Ferrence said.

Ferrence's team developed various kinds of shell scripts, new applications written in Java or C++, or used RPA as a kind of glue where it made sense. The team created a portal that lets employees with appropriate credentials trigger bots, applications and scripts as required.

Like many large enterprises, CenturyLink had legacy applications scattered throughout the business, amassed through years of corporate acquisitions. Most of these legacy scripts performed poorly but improved greatly once the developers unified everything under a singular, efficient portal.

Monitoring automation at scale

The common intake portal manages requests from business users and ultimately tracks the projects after deployment tools push them into production.

"We did not want to fund a two-month development project if we were not going to get a good ROI," Ferrence said.  

We have a lot of legacy applications where we don't have any type of interface or availability for enhancement which are perfect candidates for RPA.
Troy FerrenceSenior manager, automation solutions architecture, CenturyLink

Here's how it works. The individual business group that requests the automation project must also fund the project, and the portal organizes details to keep all stakeholders informed.  After a group requests a project, it enters the workflow of the group responsible for implementation.

Based on the description, the center of excellence teams determine whether to develop an application, write a script, implement an RPA bot or some combination of these option. The project then goes through a process assessment and creates a solution design document which specifies how it will code the automation at scale. At this point, the business can evaluate the project's financial costs and decide whether to pursue the project or not.

If the project is approved, it enters the platform, which sets up the user access management, servers and organizes the project details for the development team. Once deployed, everyone involved can track how well the project works in practice.

When business units started to see the benefits from combining RPA and other types of automation, demand for similar work accelerated across other departments at CenturyLink. Ferrence's team slowly added other federated teams dedicated to specific business units across the company.

Participants must go through RPA training and master the process required to generate documents, analyze business processes and manage reviews. The core center of excellence team developed an extensive documentation library that describes the platform, the various roles involved in a successful project and best practices to scale automation.

Scaled automation and the human factor

Even with such positive results, not everyone welcomed the initiatives. "People are not always fans of the thought of automating some of the work they do," Ferrence said.

Some projects that use automation at scale require API access or database connections to efficiently gain entry to information. Although RPA could automate these processes without breaching the APIs, it would add to the project's overhead or cost.

To work through any resistance, Ferrence said he tries to show those reluctant engineers how automation could free up some of their time to try other projects.  

The misleading nature of bot counts

Ferrence said that some managers can get stuck on the number of automations, or "bots" their group uses. He said he tries to convince managers to see the big picture, because it's more useful to focus on the number of processes and the ROI that is provided back to the overall business.

Along with a decreased focus on bots, it's also useful to adopt a mindset that looks at how to manage, deploy and scale automations, such as how to deploy microservices and containers

Down the road, Ferrence hopes to bridge this growing library of ways to scale automation with machine learning and AI tools to proactively solve problems. For example, when a technician logs a new customer trouble ticket, it would signal a back-end process to test the customer's line, identify previous orders that may have caused problems and identify relevant system-wide issues to address the problem.

Ferrence said he eventually hopes to pull statistics from the network and use machine learning to predict modem failures in areas where there are high ticket counts in response to heat or rain.  "If we know where certain areas are likely to have a problem based on the data sets, we can plan accordingly and shift our resources to have the biggest impact," Ferrence said.

Dig Deeper on DevOps-driven, cloud-native app development

App Architecture
Software Quality
Cloud Computing