Automating Labs with Python, Jinja2, and Netmiko

Following up on my last post, I have set out to start automating certain aspects of my labs. I spent a few days going over the material from Kirk Byers‘ highly-recommend Python for Network Engineers course. I studied on the previous version of his course a couple of years ago (covering Python2), but this new version, which covers Python3, is even better.

I came up with a generic topology that was purposely overengineered so that I can enable and disable links on-demand to create different logical topologies without having to interact with the physical lab topology. The lab represents a single service provider core network, multiple customer sites, and two SP-attached Internet connections. Most links will remain disabled for most lab scenarios, but are there for various cross-site, DIA and backdoor options available with this design.

To automate the baseline configuration, I created a Python script that imports the inventory from a CSV file, uses a Jinja2 template to generate the configuration for each device, and Netmiko to push the configuration to the devices. It’s kind of funny to succinctly place into a blog post something that took many hours to test and troubleshoot before coming up with the final version. The best part of gaining this kind of experience is that I can use what I have already done as a template moving forward, whether for the lab or for actual production.

The CSV file is straight-forward. The header row contains the variables for each device, such as the name, management IP, port, and interface IP addresses. Each subsequent row defines individual devices:

The Jinja2 template defines configurations for all devices, which gets populated with the individual variables, and covers device-specific configurations:


hostname {{ device }}

interface lo1
 ip address {{ lo1ip }}

{%- if ifg00ip %}
interface g0/0
 ip address {{ ifg00ip }} {{ ifg00mask }}
 no shutdown
{%- endif %}

{%- if device == 'P1' %}
int lo2
 ip address
{%- endif %}

With this example, every device is configured with the device-specific hostname. Every device is configured with a lo1 loopback address. If the device has an IP address configured for interface g0/0, the IP and mask are configured, along with making sure the interface is not shutdown. If the g0/0 IP address is not specified in the CSV file for this particular device, that configuration section is skipped. Likewise, the final section of the template will only be used if the device is ‘P1’. All other devices will skip this particular configuration section.

The Python script is the glue between the CSV file, configuration generation, and actual configuration deployment. The script imports the csv, jinja2, time and netmiko libraries. The script then defines variables for the CSV and Jinja2 files. Next, the CSV file is imported. The details of individual devices are placed into a dictionary, and each dictionary is placed into a list representing all devices. The script then generates the configuration for each device by feeding the details into the Jinja2 template. Netmiko is then used to send the output of the Jinja2 processing to the actual devices.

This kind of automation is perfect for the lab, because the CSV file represents certain baseline aspects that are not going to change, such as the IP addressing of the links between all of the service provider ‘P’ routers. The Jinja2 template can then be modified for different lab scenarios, depending on how much configuration you want to build into the baseline, per-scenario. The script could even be expanded so that it selects a different Jinja2 template based on a menu of possible scenarios. This same type of scripting setup could be used on a production network to set up new sites or push certain standardized configurations (such as enabling NetFlow on all devices). There are all kinds of possibilities.

Continue reading “Automating Labs with Python, Jinja2, and Netmiko”

Why Network Automation?

I have been wanting to get a little deeper into some various technologies surrounding MPLS and BGP-based VPNs (beyond basic L3VPN, such L2VPN, QoS, multicast, EVPN, etc.), so I assembled a virtual lab with approximately 30 routers which represent a service provider core and several “customer” sites, along with two sources of fake Internet connectivity (or more accurately, a simulated Default-Free Zone (DFZ)). After I earn a deeper understanding of topics within a single service provider core, I will expand this to inter-provider topics. Yes, I meant “earn”, since more work will be involved beyond just reading.

I was getting ready to develop an IP addressing scheme for the core network, and I realized I have a good opportunity here to get deeper into network automation. While studying for the CCIE R&S lab, I spend quite a lot of time in a text editor building configurations to review before pasting them into the device consoles. For tasks that involve repetitive configurations, copy-and-paste is my friend. You don’t (yet) have access to anything like Python or Ansible in the CCIE R&S lab to try to automate things (though I suppose you could use TCL if you really wanted to).

A good portion of setting up a large lab environment of any kind is developing and applying the baseline configuration. I’ve done this countless times over the years, and I was getting ready to do it yet again when it occurred to me that if I invest some time now to develop and use some network automation processes, the buildup and teardown of future labs will be so much quicker. I’ve dabbled with this in the past; I learned the basics of Python and have developed a few network-oriented scripts. I found I enjoy working through the logic and seeing the working results. I also developed and deployed a simple Ansible script to push out some configurations on my current production network.

I read the mostly-complete “rough cuts” version of Network Programmability and Automation by Jason Edelman, Matt Oswalt, and Scott Lowe. This is a really fantastic book, and along with Russ White and Ethan Banksnew book, I consider it an absolute must-read for anyone wishing to progress their career in computer networking and establish a very strong set of foundational knowledge (I swear I’m not trying to name-drop, but I’ve read a LOT of networking books, and these really are toward the top of the list). When I read Network Programmability and Automation the first time, I used the knowledge as an overview of some of the things that are possible within the realm of network automation. Now I’m going through it again to develop the skills necessary to automate the deployment of my lab configurations.

One thing I believe hinders many people wanting to dig deeper into automation (myself included), is having a use case. It’s easy enough to say that if you have to do a single task more than once, you should automate it. Automate all the things, right? There are two issues I see here: the underlying knowledge of what it is you’re trying to automate, and the ability to evaluate the “bigger picture”.

For example, within the context of networking, you could learn how to automate the deployment of complex configurations for a particular routing protocol, but what good is that going to do if you don’t fully understand what those configurations mean? Automation presents you with an opportunity to make mistakes much faster, and on a much more grand scale. If you automate configurations among several devices and things end up not working as anticipated, can you troubleshoot what went wrong?

Likewise, evaluating the bigger picture helps to understand where particular forms of automation are helpful, and where you will run into diminishing returns. For example, you could automate just about every process involved in network configuration, but nearly every business is going to have exceptions that need to be handled somehow, and automation may not be the answer in those instances.

Tying both of these concepts together, I realized the opportunity to automate the things I know extremely well due to my previous knowledge and experience, such as baseline configurations involving hostnames, IP addresses, basic routing protocol configuration, etc. Because I know how all of these things work very well, I can easily automate this as well as troubleshoot potential issues if things don’t go as expected. In the bigger picture aspect, the purpose of the lab is for me to understand other topics that use the baseline configuration as a prerequisite, and therefore I am not yet ready to automate those technologies because I do not yet have a full understanding of their nuances.

In other words, the more you learn, the more you can automate. You need to develop skills on how to automate things, but if you automate things you do not understand, you are setting yourself up for future frustration. Don’t let this discourage you from learning and getting deeper into both automation and classical network engineering skills. Increasingly, the two go hand-in-hand, but you can certainly end up in a chicken-or-the-egg scenario. My advice is to “earn” the networking skills first, and automate them second.

Mind Map for CCIE & CCNP Routing & Switching

I created a mindmap of topics that are covered on the current Cisco CCIE RSv5 lab exam to help myself study, and I thought my work might be useful to the general network community as well. I included CCNP R&S in the title, because there’s a lot of overlapping information that I think most people pursuing the CCNP might find useful as well. I have covered a lot of topics by way of configuration examples that I remember struggling with a little bit when I was studying for the CCNP (wow, it’s been five years already!).

This document contains a hierarchy of topics with their associated Cisco IOS configuration syntax. These commands should work on most versions of classic IOS and IOS-XE, versions 15 and later. I tried to be as comprehensive as possible with regard to the covered topics referenced against the current CCIE R&S blueprint, however it is next to impossible to truly cover every configuration aspect within a single document, mostly because any given topic set (and more) may be covered in any specific delivery of the lab exam. In other words, you won’t really know until you get there.

This document provides the configuration syntax for nearly all topics covered. In many cases, examples and explanations are also provided, but not for every single topic (you still need to do your homework first!). Likewise, verification commands are generally not included here, because that would have easily doubled or tripled the size. For any topic you wish to know more about, I highly recommend looking at the official Cisco documentation. Most topics have both explanations as well as configuration examples there. Much of what is contained in this document was sourced from the official Cisco documentation.

This document is NOT:

  • A hand-holding guide through all CCIE topics
  • Any sort of answer key for any sort of specific lab scenarios
  • A comprehensive guide to every possible topic you might encounter on the lab exam
  • A replacement for pretty much any other form of studying

That being said, this document can serve as a good supplemental quick-reference to the vast majority of topics on both the CCNP and CCIE Routing & Switching exams. CCIE topics are limited to the lab blueprint only (I didn’t cover IS-IS, for example).

The original version of this document will always be available here. It was created with the excellent and highly-recommend MindNode software for the Mac. I have included the original MindNode file here, and several other formats as well. The original MindNode file lets you expand and collapse branches of the tree as desired.

Some browser plugins may have trouble viewing some of these files. If a file does not display properly in your browser, try downloading the file and opening it with a different application.

Don’t forget about my 3500 CCIE flashcard deck, and my blueprint documentation reference guide. If you found this to be useful, please let me know on Twitter or LinkedIn. Thanks!

CCIE FlashCard Deck for R&S v5.1

Today I have decided to finally release the flashcard deck that I created for myself in order to pass the written portion of the CCIE Routing & Switching v5.1 exam. This deck represents many months and hundreds (if not thousands) of hours of study effort.

After passing the exam, I had considered putting together some kind of study package, but I recently started thinking about how version six of the CCIE R&S is going to be here before too long, and it serves no purpose for me to sit on this accumulated knowledge and not share it.

That being said, I still believe you will receive a greater benefit by creating flashcards yourself, as I’ve written about previously. The process of comprehending what you are studying and converting that knowledge into a useful format for yourself is very empowering. Sometimes, though, it helps to get an outside viewpoint, which is what this deck of nearly 3,500 flash cards represents.

I passed the written exam using these cards I created. However, there are most certainly different errors and mistakes present in the cards. This could be due both to my sources having errors, as well as from me misinterpreting something. Or it could be as simple as a type-o.

Nearly all cards are tagged with at least one topic. My flash card deck is available in the following:

I hope this proves to be useful to you! Thank you.

Bonus: Document I assembled in 2017 containing reference links for most blueprint items. Includes direct links in the CCIE OCGs on Safari, direct links to official Cisco documentation, particular Cisco Live sessions, and more.

Retrospection and the Future: Still Studying for the CCIE Lab

Last October marked the five-year anniversary of my blog. I had started an introspective post looking back over the past five years, but the truth is, I feel like I spend more time looking forward than backward at this point.

That’s not to say I haven’t come quite a long way in the past five years. I sometimes need reminding of that whenever I feel like I’m progressing too slowly. Occasionally, I get down on myself for being in my late thirties and still toward the beginning of my career in networking, when I see many shining examples of people much younger than me who are much further along in their careers. However, I cannot change the past and my various circumstances, and I cannot go back and make different decisions at different points in my life.

I can recognize, though, that just five years ago, I was considered a high-end technician or perhaps low-end systems administrator (even though I had some light networking experience sprinkled in there as well). Since then, I progressed into enterprise and networking support where I received a senior-level promotion in less than a year, then to a full-on network engineer, and now to senior network engineer with another recent promotion. I occasionally need to step back and realize that, regardless of age, there are people who never make it as far as I have, let alone in just five years. My wonderful wife is always great at reminding me of these things whenever I need a pickup.

So, to looking forward: my last post detailed passing the CCIE R&S written exam, which was a very big milestone for me since I’m doing this out of self-interest. However, the lab exam is still a decent amount of time and work away from where I am now. I still spend a large amount of time each week studying for it. Having access to the Cisco Expert Level Training has been huge for me. It’s no magic bullet, as achieving the CCIE legitimately requires a LOT of personal work and dedication, but I feel like it helped greatly to organize and narrow down the topic scope into a more manageable form of study.

I stated before that there is a huge difference between the written exam and the lab exam, and that it probably is in fact a good idea to study for the lab first, then pass the written exam shortly before you plan to attempt the lab. I also stated, though, that for me, passing the written so early was an important method of self-validation. That being said, I found out very quickly that there is a huge difference between just knowing the technologies, and being able to configure them.

Going through the first few CELT workbook labs, there were several instances where I knew exactly what they were talking about, but I either could not remember how to configure them, or I couldn’t solve the task in the exact way for which they were looking. I also quickly learned that I needed to improve my reading comprehension as well as attention to detail.

On top of that, there are things that simply require outright memorization for the sake of speed during the lab. To that end, I’ve been digesting the material a little differently than I did for the written exam. With the written exam, I made and studied detailed flash cards, eventually creating a deck of nearly 3,500 cards before I took and passed the exam on the first attempt. With the lab exam, I’ve been going over various materials yet again, going even deeper where necessary, and creating new study materials for myself.

The first thing I did was create a mind-map in several phases. In the first phase, I created high-level topic domains following the hierarchy of the CELT program (Layer 2, IGPs, BGP, MPLS, etc.). For the second phase, I went through every item on the official lab blueprint and molded the topics into the appropriate places within the appropriate hierarchies in the mind map. I then used INE’s expanded blueprint to fill in the remaining topics (pruning the list where necessary – not everything on their expanded blueprint is actually on the lab exam). Finally, I covered the specific configuration of every topic in the hierarchy by going through the configuration guides and command references, and cross-checking it with the version of IOS used on the lab.

I then worked on expanding my Python skills a little bit by creating a set of scripts that take each of the topic items and randomizes certain elements, such as names, interfaces, ASNs, etc. The idea was to create something similar to flash cards, but to make it less repetitive since the elements are randomized wherever possible. As I write this, the script is still very simple in that it presents you with an isolated task, you enter the answer, and then it shows you the correct answer. I have not yet progressed into combining topics together for questions.

You might wonder what value this has for me personally since I am the one who created all of the questions. The CCIE is absolutely massive, and it covers a lot of technologies that I may never actually work with in production, and therefore it is difficult to memorize everything. There are many topics where I know how they work and what they are supposed to do (and what problem they are supposed to solve), but remembering the exact syntax offhand can be difficult.

One of the features I intend to code into the script is the ability to judge how difficult the question was (similar to Anki), so that in future sessions, I can simply skip over the easy ones and drill in the harder questions. I may also work the script into a GUI or web-based version in order to further expand my Python skills, but at the same time I’m doing my best to not take my eye off the prize, so to speak, and make sure that I’m not being distracted too much from actually studying for the CCIE.

Distractions can be very difficult. Learning scripting and automation is important for network engineers. However, my current work environment does not require it, so I’ve only dabbled in it here and there just to make sure I understand the concepts. I imagine I will immerse myself deeper into it (and other topics as well) after completing the lab exam.

I have not had my sole focus on studying for the lab, which has been making it take longer to achieve. I do not intend to stay within enterprise networking forever, and I have been introducing myself to various service provider topics. After passing the CCIE, I intend to try to find work within the realm of service provider networking.

However, I keep reminding myself that passing the lab is a personal goal, not a professional requirement. This can be both an advantage and a disadvantage, depending on how I am feeling at any particular moment. The advantage being that I need to remind myself occasionally not to put too much pressure on myself. The disadvantage being that there are so many things I want to explore, but the CCIE requires dedication of some form.

When the studying gets tough, I question the value of the CCIE, as I’m sure everyone who has ever studied for it also has. This of course has been covered in every imaginable outlet with every imaginable argument both for and against since the inception of the CCIE in 1993. I can see both sides of most arguments. I am still taking the approach that simply studying for it, whether or not I ultimately obtain the trophy, is making me a better network engineer. I can much more comfortably solve problems now using different methods that I would not have been able to just a couple of years ago. And I can do it in a vendor-agnostic way when necessary. It’s certainly true that you don’t need any certifications to be able to do that. But once again, the certification has provided a meaningful learning path for me to follow.

It’s difficult when you’re studying topics that you know you’ll never actually work with in real life, or when working with normal technologies and having to push them into types of configurations that would probably never occur on a production network. But I keep telling myself that working with these various technologies in these various manners will help me to recognize patterns in the future, and ultimately make me a more seasoned engineer.

I still have at least a year worth of work before I plan on taking the lab exam. I am doing my best right now to drill in the basics so that if I need to refer to the documentation, it will only be for those rare miscellaneous topics, if at all. I strongly believe this initial effort, while slow, will pay off in the end.