NSX Advanced Load Balancer - Logging and Troubleshooting Cheat Sheet
Get into the OS Shell (all elements)
sudo su
Controller Log Locations
Note: Everything in /var/lib/avi/logs
is managed by Elasticsearch. I wouldn't mess with it.
Events published to the GUI: /var/lib/avi/logs/ALL-EVENTS/
The primary log directory for Avi Vantage Controllers is /opt/avi/log
. As this feeds into Elasticsearch, they have file outputs for every severity level. An easy way to get data on a specific object would be to build a grep
statement like this:
grep {{ regex }} /opt/avi/log/{{ target }}
-
alert_notifications_*
: Summarized problems log. Events are in ajson
format!
Troubleshooting Deployment Failures
-
avi-nsx.*
: Presumably for NSX-T integration. further investigation required -
cloudconnectorgo.*
: Avi's cloud connector is pretty important given their architecture. This is where you can troubleshoot any issues getting a cloud turned up, or any initial provisioning issues. -
vCenter*
: vCenter write mode activity logs. Look here for SE deployment failures in a traditional vSphere cloud.
Service Engines
Troubleshooting
Checking the Routing Table
NSX ALB / Avi uses FRRouting (7.0 as of release 20.1) over network namespaces to achieve management/data plane separation and VRF-Lite. To access the data plane, you will need to change namespaces! Unlike NSX-T, this doesn't happen over docker namespaces. This means that the follow commands work in both as root:
- Show all VRF+Namespaces
ip netns show
- Send a one-shot command to the namespace:
ip netns exec {{ namespace }} {{ command }}
Example:ip netns exec 'ip route show'
- Start a shell in the desired namespace:
ip netns exec {{ namespace }} {{ shell }}
Example:ip netns exec avi_ns1 bash
After in the bash
shell, all normal commands apply as if there was no namespace/VRF.
For more information on Linux Network Namespaces, here's a pretty good guide: https://www.opencloudblog.com/?p=42
Logging
All SE logging is contained in /var/lib/avi/log
. Here are the significant log directories there:
-
IMPORTANT!
bgp
: This is where all the routing protocol namespace logging from FRRouting lands. -
traffic
: This one's pretty touch to parse and it's better to use Avi's Elasticsearch instead.
Conclusion
Avi Vantage has a pretty solid logging schema, but is very much a growing product. These logs will eventually be exposed more fully to the GUI/API, but for now it's handy to grep
away. I'll be updating this list as I find more.
Top comments (0)