The Network Automation Forum was founded this past summer by Chris Grundemann and Scott Robohn to answer the question: “Why haven’t we seen full adoption of network automation yet?”

And this past Monday and Tuesday in Denver, they hosted autocon0 to gather several hundred of the industry’s greatest minds to try and do just that.

So, are we any closer to an answer?

I’m not sure, but it sure was a great couple of days! These guys stuffed us full of information and I’m sure I’ll be digesting it for weeks to come, but my initial takeaways are:

Don’t try to boil the ocean, just do something.

I think it was Mark Ciecior of Carrier Access IT who said something like this, and several others echoed these sentiments. I guess this is what I, too, have been trying to say with my Four Stages of Automation and constant references to it in all my other posts–there are many ways to get started, just don’t make it a huge monolith. Chip off a chunk of rock and get to work, get a small win and build on it over time.

We have to cross disciplines to get this done.

Generally, you will either need to train software engineers on networking, or network engineers on coding and best practices like CI/CD, because there are not many out there yet to hire who have both (CU Boulder Network Engineering grads excepted).

  • Those automating brownfield environments found that many network engineers are resistant to or even fearful of all the new concepts, and so extending lots of help and guidance is key.
  • Garret Nowak of 11:11 Systems shared the idea of Automation Fridays, where they spend a chunk of the afternoon trying a new language, developing a new tool or just asking and answering questions.
  • He also discovered that the network engineering team ended up with half the time to get the usual things done, because they were spending all this time on the new automation tasks.

Which leads to the next takeaway:

It takes buy-in from the entire organization.

I’ve run into this in the past myself, the flip side of the “just get started” gem above. If network engineering automates a few things because it’s cool or because it saves them time on repetitive tasks, that’s great! Without support though, or maybe even enthusiasm, from the rest of the organization, it’s not likely to last or scale. Why?

  • It takes a long time to write code, when you can “just” use the CLI in a fraction of the time to do that one thing, so your intial delay in getting results can frustrate others.
  • Management usually needs to see a concrete benefit, or they won’t allocate the resources (time, vendor expenses, etc.), to maintain or scale what you’ve built, and
  • What you’ve already built will tend to sit unused if ops and engineering staff aren’t aware of or familiar with the environment or tools needed. Some of this can be helped with the extra guidance mentioned before, but with lots of fires to put out, CLI will often remain the default without executive sponsorship.

So, it sounds like you need to have a business case, but:

Your business case depends on your business model

At lunch yesterday, Tuesday, there were so many great Birds of a Feather topics it pained me to have to choose just one, but I’m glad I ended up at the Business Case table after all. Among the insights:

  • Huge cost savings can be garnered by integrating various SOTs including AP/AR systems and reconciling the differences with DCIM—some companies are unaware they are paying, sometimes for years, for cross-connects or power feeds they’ve never even activated.

  • If you run a network as a business, i.e. you derive your revenue from your network, you can probably make a case for automating based on cost savings, MTTR reduction and provisioning efficiency—specifically, to limit the revenue lost every day a customer has to wait for service. But if your network is incidental to your main business, like retail, then you may need to make a case for security, because transactions are probably your bread and butter, or the ability to open new stores more quickly.

  • One automation vendor mentioned that not a single sale has been made on headcount reduction. Some organizations, however, do want to limit incremental headcount while agreeing that nobody wants to have a goal of reducing existing staff.

There’s not a lot of agreement on build vs. buy

There were vocal proponents of both, and there were those who laid out the pros and cons.

  • If you build, you may be duplicating work and it’ll take you a long time (and thus necessitate headcount and cost), but you’ll probably get a solution that’s tailor-made for your network.

  • If you buy, it saves time in the long run but can a) be very expensive and b) be a bad fit or lacking in flexibility.

  • Some say you can have both: buy something, or several things, and build the tool that makes up the difference or ties them all together. Many open source maintainers allow you to buy support, so this is possibly another middle ground.

The Upshot

There was so much good content in the presentations and panels beyond what I’ve already touched on, such as the concept of version control for the SOT database or batch-scripting troubleshooting commands to generate TAC cases, and I can’t remember many conferences where so much was so relevant to me and held my attention for so long.

There was also a huge turnout, and every single person I encountered was impressive. I saw some old friends and made many new ones, and some I had hoped to meet I never ran into, even though I saw them on the stage. More than once I heard it remarked that the level of knowledge and insight there was unprecedented compared to other conferences. I know I myself didn’t have a single conversation that wasn’t thought-provoking or inspirational in some way, and all of it was so engrossing I didn’t remember until late Tuesday to take some selfies.

Huge thanks to Chris and Scott for pioneering such a groundbreaking and fun event–I can’t wait for the next one!

Selfies with me, in various states of blur, arm bunching and unreadiness, and (L-R by row) Cat Gurinsky, Colin McIntosh, Mau Rojas, and Urs Baumann.