Attempting to Avoid Obsolescence

This post is pretty long and meandering. The crux of it is that I wrote and used my first Python script on a production network yesterday, and it made me pretty damn happy to see the results.

Now for the Director’s Cut:

I’m sure the title represents something I will be forced to revisit many times during my career, and we all know that everything old is new again (Rule 11, naturally). As I get deeper into my career and expand both my experience and base of knowledge, I see Rule 11 all the time. When you’re first starting out, you may be aware of the concept, but it definitely takes time and experience to truly appreciate it.

A couple months ago, I started looking more into scripting and network automation. I mentioned Rule 11 because I am well aware of the fact that, despite all the industry buzz during the past couple of years, this is not in any way new. Before Python took over as the scripting language of choice for network engineers, it was Perl.

However, scripting seems to remain a relatively important skill for network engineers dealing with systems of any sort of scale. Many of the end results of scripting can be simplified by purchasing expensive network management software. I have discovered, though, that even with the expensive NMS software in place, sometimes you would like to gather very specific information, and either the NMS doesn’t support what you’re looking for, or it is extremely unintuitive and/or cumbersome as to how to actually obtain the desired information from the software.

I enjoy learning the “classical” network engineering skills, such as the various routing and bridging protocols, and architecture and design. That is one of the motivations for me still studying for the CCIE. But, as I’ve written about in the past, it’s taking me a long time to get the CCIE because that is not my sole focus. I don’t want to be a one-sided engineer who can’t fathom thinking outside the world of Cisco, and I don’t want to be the kind of person that designs networks a certain way because “that’s how it was on the Cisco certification exam.”

On the other side, I’m not necessarily interested in being the “full stack” engineer, either. There are many sysadmin duties and responsibilities that I am glad I do not have (like worrying about people’s files and email). Yet, I know enough about Windows and Linux to do the things that I need to do. I know the basics of wireless networking and IP telephony. I have a decent level of VMware vCenter knowledge and experience (mostly through breaking and fixing things in my home lab). I also have some knowledge of SANs and storage systems, though I will also admit that I find the networking aspect of SANs very appealing and may explore that at a deeper level later on.

Still, there’s enough industry talk about scripting and network automation that I decided it was time to investigate a little on my own. I will very readily admit that the idea of working with APIs where you can send and receive actual, exact information, as opposed to what you get with screen scraping, is very appealing. It’s going to take awhile before I reach that point, especially considering the vast majority of equipment I work with currently is 10-15 year-old classic Cisco IOS-based. That means interaction via screen scraping.

I started my current job a little more than a year ago. While my last job gave me the absolute tiniest taste of enterprise-level network engineering, my current job has been full-on, giving me so much of the experience I have desired for the past several years. While I may be working with primarily older technology (there’s some brand new stuff sprinkled in here and there), it is at a scale that is large enough where certain automated tasks begin to make sense.

When I started, we had no configuration backup process in place for the routers and switches (that I was aware of). I discovered RANCID, and built my own server from scratch. I had never worked with this software prior, and while we are using SolarWinds Orion for some things, we are missing the NCM piece, which I discovered was ungodly expensive.

Figuring out how to set up and use RANCID has been one of the best things I’ve done so far for myself. I have used it so many times for various things, and it has been a real time saver on occasion. Just the other day, I had to send out a replacement router, only I didn’t find out about it until 20 minutes before UPS was scheduled to do their normal pickup. By having the configuration already backed up in RANCID, I was able to quickly grab a router, erase its config, put the backup config on it, and get it packaged and shipped out within 20 minutes before the deadline.

I experimented with sending out certain commands to all of the routers via RANCID, and that was pretty neat, but felt kind of awkward for some reason. I knew that I would be able to do more if I learned how to script with something like Python. A couple months ago I started making my way through Learn Python the Hard Way, and I’ve also made it halfway through Kirk Byers‘ excellent Python For Network Engineers course.

I found learning some of the stuff pretty difficult at first. Part of it was me doing my best to let go of my attitude regarding programming. I got my Bachelor’s degree in Information Systems, not Computer Science, because I was more interested in networking, and not at all interested in programming. But as time goes on, I feel like I made the wrong choice. Not because I want to be a programmer now, but because having that background could have been more beneficial to me for the future. However, I am sure that writing scripts is not at all like being a full-on software developer so I should probably not make that comparison.

Right now I am still at the extreme beginner stage, and I know that I will need to go over the learning material several times if I wish to reach the point of being able to “think in code”. Still, I knew that I would need to start somewhere if I wanted to make what I am learning try to stick. Sometimes, the hardest part of learning is trying to see where and how what you’re learning can be applied. Coding is all about breaking down problems into the smallest amounts possible, and then recombining them into something meaningful.

This is something that can be very difficult to see at first. “I want to learn Python, but what do I want to do with it?” Beginning with the knowledge that you can do practically anything with it, as long as you know how, does not help at all. I needed to start smaller. I needed a single simple task that I could start with, and learn how to build upon it as needed. During this past week, I had been thinking about this more.

I wanted to start with a simple script that could take a list of hostnames or IP addresses of Cisco routers, send a command to them, and dump the results to a text file. On the surface, that sounds easy enough. Yesterday, my boss wanted me to gather information about how many of our branch offices had more than six Cisco phones, but did not have Cisco switches installed. I saw this as the perfect opportunity to finish what I had started and have an actual use case for the script.

Ultimately, the script would need to log into each specified Cisco router, issue a command (“sh cdp n | inc SEP” in this case), and dump the results to a text file. I was able to cobble together a script based on what I had learned from the first several chapters of Learn Python the Hard Way, the first half of Kirk Byers’ course, and a couple of quick web searches on how to write code for a couple of specific tasks. I created a GitHub account a few months ago, and as embarrassing as my first script might be, I decided to go ahead and post it to my account anyway. If nothing else, I figured it was something else that I could say I have done.

The first issue I ran into was scope creep. Should the script do this? Should it do that? It would be nice if it did this other thing. I saw how this could very quickly get out of control. I realized that, for now, I am still learning, and I just need to accomplish this simple single task at the moment. Keep it simple.

The second issue I ran into was errors I didn’t anticipate and wasn’t sure how the script would handle them. The first thing I experienced was when a host was not reachable. I knew from the start that this would be an issue, but I wasn’t sure how to handle it. Through a web search, I found out how to ping a host, and then return a True or False. I used this as the primary error catch.

Other errors I ran into, but have not yet resolved, include bad authentication and command timeout. Sometimes, the two are related. Upon reaching either of these failures, my current script just dies and does not handle the exception. This will probably be the first thing I try to improve upon.

Going back to the idea of scope creep, as I was putting the script together, I was thinking of all kinds of different ways that the script could be added to and enhanced…if only I knew how to do it. For example, in my current use case, I only wanted to know which locations have more than six Cisco phones, but do not have a Cisco switch. If I knew Python a little better, I could write a script to scour the network and present back to me only this exact information, instead of the extraneous information provided by screen scraping. However, this is something that will develop over time, because learning this stuff is just one of many skills I wish to gain experience in, and it all takes time.

In the end, the script did what I needed it to do, and gathered the information that I required. I was able to get the information I needed by letting the script run over a lunch break, whereas my alternative (getting information from our IPAM system) would have taken much longer, due to the specific information I needed. I was very excited by this, and I can see this turning into something important over time.

There’s a lot out there to learn, and it is very difficult sometimes to remain focused on any single thing. Many times, what you learn is dictated by the immediate business problem you need to solve. Rarely is what you need to learn singularly-focused, and learning it all takes time, and practice. It can be a delicate balance to not be pulled into too many directions at once, both in real life, and in “study life”. I’ll admit, that in itself is another skill I am still trying to learn.