Running deployment scripts on Cisco routers @ AWS from a Private Github repository

Background:

I've been building with a client of ours their landing zone, and for the network connectivity part Cisco routers were selected to be used. This would connect nicely to their existing on premises network.

"Ok, sounds easy enough. Let's automate this fully."

The last sentence gave us a challenge, the documentation is non existent, and in the background Cisco does their automation - which is lacking features and not really working as one would expect.

We worked closely on this with Christofer on this, weeks bashing our heads to the wall - utilising support channels, submitting bug report, waiting.

So I'll open up here our experience, and hope it helps someone else sometime in the future building similar solution. If you just want the copypaste for your user data head to Solution part at the bottom.

Disclaimer I'm not a Cisco specialist, all my thoughts here come from usability perspective utilising AWS cloud.

Starting situation:

1) Deployment script (deploy.sh) for setting up the routers is in Github

• This script generates everything needed to configure the router, tunnels, VRFs, so forth. It utilizes instance metadata for required information. I won't go into detail on this, this was written by clients network engineer.

2) Token has been created to access GitHub
3) BYOL model is used for the licences, AMI from market place that we used is aws-marketplace/Cisco-C8K-17.06.01a
4) Secrets are stored in Secrets Manager

The story:

Cloudformation was written, to build the basic infrastructure required. Nothing special there.

I'll focus on the instances user data parts, that's the main pain point.

For licensing and installation of AWS CLI and HA package, added to user data:

      Section: license
      TechPackage:appx

      Section: Python package
      csr_aws_ha 3.1.0
      awscli 1.20.40 sudo

The licence command fails, and tells you that correct option is ‘vacs’, ‘lite’, ‘ipbase’, ‘ax’, ‘security’ or ‘appx’ - wait what? - this is a bug and will be fixed by Cisco in a later AMI.

In CSR1000v these were correct options, but in C8000v options are ‘network-premier’, ‘network-essentials’ or ‘network-advantage’.

You'll also need to configure a IAM role for the instance, check requirements for the HA script from Cisco.

Then to run the deployment script. :)

We tried to get the script from Github using builtin options.

Few screenshots of Cisco bootstrapping manual:

So we did what was asked, tested with curl that our URL works, and set user data as:

      Section: scripts
https://token@raw.githubusercontent.com/Owner/Repository/main/deploy.sh

As you guessed this wouldn't work. Cisco actually in the background utilizes wget for all https requests - which in turn doesn't support tokens in the URL. Curl is only utilised for ftp.

So we tested, wget from GitHub works if you give it --user whatever and --pass token

Lets try this

So added credentials - no luck, only works for FTP, not HTTPS.

We wen't back and forth, testing that everything works if when we have our deployment code in a public website.

Considered using a wrapper script, hosted publicly. Which would do three things:
1) Get token from Secrets manager
2) Download deploy script with curl
3) Run deployment script

Tried to run that though Section: scripts and I think that actually worked, but since there was two minds on this was dropped when we found another solution that works.

Noticed from the logs that Cisco actually themselves use event manager applet while booting up to run the user data, why not do that ourselves also?

Solution

Section: IOS configuration works nicely, we can run your commands there - utilising event manager applet we ran the deploy script.

      Section: IOS configuration
      event manager applet Deploy authorization bypass
      event timer watchdog time 180 maxrun 360
      action 0010 cli command "enable"
      action 0015 syslog msg "Getting the secret"
      action 0020 cli command "conf t"
      action 0021 cli command "do guestshell run aws secretsmanager get-secret-value --secret-id github/access-token --region eu-west-1 --query SecretString --output text"
      action 0022 cli command "event manager environment _secret $_cli_result"
      action 0023 cli command "end"
      action 0030 syslog msg "Downloading the deploy-code"
      action 0031 syslog msg "guestshell run curl https://$_secret@raw.githubusercontent.com/Owner/Repository/main/deploy.sh -o /home/guestshell/deploy.sh"
      action 0035 syslog msg "Running deploy.sh"
      action 0040 cli command "guestshell run bash /home/guestshell/deploy.sh"
      action 0100 cli command "conf t"
      action 0110 cli command "no event manager applet Deploy"
      action 0115 cli command "end"

So that's it, we got keys from Secrets manager, downloaded our script to generate the config and removed the applet afterwards.

Deploy script also has builtin checks that it won't run twice.

One last note on running scripts within Guestshell

As a note, for your scripts that you run in guestshell that while #!/usr/bin/env python is a valid script interpreter #!/usr/bin/env bash is not, you have to use #!/bin/bash there.