Automating Labs with Python, Jinja2, and Netmiko

Following up on my last post, I have set out to start automating certain aspects of my labs. I spent a few days going over the material from Kirk Byers' highly-recommend Python for Network Engineers course. I studied on the previous version of his course a couple of years ago (covering Python2), but this new version, which covers Python3, is even better.

I came up with a generic topology that was purposely over-engineered so that I can enable and disable links on-demand to create different logical topologies without having to interact with the physical lab topology. The lab represents a single service provider core network, multiple customer sites, and two SP- attached Internet connections. Most links will remain disabled for most lab scenarios, but are there for various cross-site, DIA and backdoor options available with this design.

To automate the baseline configuration, I created a Python script that imports the inventory from a CSV file, uses a Jinja2 template to generate the configuration for each device, and Netmiko to push the configuration to the devices. It’s kind of funny to succinctly place into a blog post something that took many hours to test and troubleshoot before coming up with the final version. The best part of gaining this kind of experience is that I can use what I have already done as a template moving forward, whether for the lab or for actual production.

The CSV file is straight-forward. The header row contains the variables for each device, such as the name, management IP, port, and interface IP addresses. Each subsequent row defines individual devices:

The Jinja2 template defines configurations for all devices, which gets populated with the individual variables, and covers device-specific configurations:

hostname {{ device }}

interface lo1
 ip address {{ lo1ip }} 255.255.255.255

{%- if ifg00ip %}
interface g0/0
 ip address {{ ifg00ip }} {{ ifg00mask }}
 no shutdown
{%- endif %}

{%- if device == 'P1' %}
int lo2
 ip address 2.2.2.2 255.255.255.255
{%- endif %}

With this example, every device is configured with the device-specific hostname. Every device is configured with a lo1 loopback address. If the device has an IP address configured for interface g0/0, the IP and mask are configured, along with making sure the interface is not shutdown. If the g0/0 IP address is not specified in the CSV file for this particular device, that configuration section is skipped. Likewise, the final section of the template will only be used if the device is ‘P1’. All other devices will skip this particular configuration section.

The Python script is the glue between the CSV file, configuration generation, and actual configuration deployment. The script imports the csv, jinja2, time and netmiko libraries. The script then defines variables for the CSV and Jinja2 files. Next, the CSV file is imported. The details of individual devices are placed into a dictionary, and each dictionary is placed into a list representing all devices. The script then generates the configuration for each device by feeding the details into the Jinja2 template. Netmiko is then used to send the output of the Jinja2 processing to the actual devices.

This kind of automation is perfect for the lab, because the CSV file represents certain baseline aspects that are not going to change, such as the IP addressing of the links between all of the service provider ‘P’ routers. The Jinja2 template can then be modified for different lab scenarios, depending on how much configuration you want to build into the baseline, per-scenario. The script could even be expanded so that it selects a different Jinja2 template based on a menu of possible scenarios. This same type of scripting setup could be used on a production network to set up new sites or push certain standardized configurations (such as enabling NetFlow on all devices). There are all kinds of possibilities.

Here is a generic version of my script:

#!/usr/bin/env python3
import csv
import jinja2
import time
from netmiko import Netmiko

## Netmiko debugging
# import logging
#logging.basicConfig(filename='netmiko_output.log', level=logging.DEBUG)
#logger = logging.getLogger('netmiko')

csv_file = 'hosts.csv'
jinja_template = 'template1.j2'

inventory = {} # Overall working dictionary
inv_list = [] # This list will contain each device as a dictionary element

# Parse the CSV file
with open(csv_file) as f:
    read_csv = csv.DictReader(f)
    for vals in read_csv:
        # Read in all k/v's per row. I'm sure there's some real cool
        #  programmatic way to do this
        # These values must match the header row in the CSV file
        inventory['device'] = vals['device']
        inventory['host'] = vals['host']
        inventory['port'] = vals['port']
        inventory['lo1ip'] = vals['lo1ip']
        inventory['ifg00ip'] = vals['ifg00ip']
        inventory['ifg01ip'] = vals['ifg01ip']
        inventory['ifg02ip'] = vals['ifg02ip']
        inventory['ifg00mask'] = vals['ifg00mask']
        inventory['ifg01mask'] = vals['ifg01mask']
        inventory['ifg02mask'] = vals['ifg02mask']

        # Build a list with each row as a dictionary element
        inv_list.append(inventory.copy())

# Generate the configurations and send it to the devices
for items in inv_list:
    print('\nCurrent device: ' + items['device'])

    # Generate configuration lines with Jinja2
    with open(jinja_template) as f:
        tfile = f.read()
    template = jinja2.Template(tfile)
    # Convert each line into a list element for passing to Netmiko
    cfg_list = template.render(items).split('\n')  

    # Connect directly to device via telnet on the specified port
    conn = Netmiko(host=items['host'], device_type='cisco_ios_telnet',\
           port=items['port'])

    # Check if device is in initial config state
    conn.write_channel("\n")
    time.sleep(1)
    output = conn.read_channel()
    if 'initial configuration dialog' in output:
        conn.write_channel('no\n')
        time.sleep(1)

    # Send generated commands to device
    output = conn.enable()
    output = conn.send_config_set(cfg_list)

    # Display results
    print('\nConfiguration applied: \n\n' + output)

    # Probably a good idea
    conn.disconnect()