DEV Community

Alexey Melezhik
Alexey Melezhik

Posted on

Super flexible infrastructure audit with Outthentic

#perl #linux #testing #devops

https://raw.githubusercontent.com/melezhik/outthentic-dev.to/master/images/outthentic-example.png

Bringing devops to our daily life makes creation of new servers is really simple. Cloud providers makes it possible to create new virtual machines in just one click. So the importance of the server infrastructure audit tools become more and more important. Tools like goss and inspec do the job.

Outthentic, being a universal script runner can do that too, but in very different way

Instead of providing declarative DSL to check server state, Outthentic supplies us with effective primitives to parse command output and validate server resources and configurations.

In some real life examples I will show you how, and if you find this approach attractive, welcome to our camp (;

Install Outthentic

Before creating any test let's install Outthentic.

$ cpanm https://github.com/melezhik/outthentic.git
Enter fullscreen mode Exit fullscreen mode

Now let's write tests

Check if a package is installed

Let's start with something simple, yet useful example. Here we just run bash command and check that exit code is 0.

$ nano story.bash

yum list python 2>/dev/null | tail -n 1
Enter fullscreen mode Exit fullscreen mode 
$ strun

2018-10-01 20:53:25 :  [path] /
python.x86_64                      2.7.5-69.el7_5                       @updates
ok  scenario succeeded
STATUS  SUCCEED
Enter fullscreen mode Exit fullscreen mode

Checking of minimal package version would take just an adding a few lines to a check file:

$ nano story.check

regexp: python.x86_64\s+(\d+)\.(\d+)\.(\d+)-
generator: <<CODE
  [
    "assert: ".( capture()->[0] >= 2   || 0  )." major version is more or equal 2",
    "assert: ".( capture()->[1] >= 7   || 0  )." minor version is more or equal 7"
  ]
CODE

$ strun

2018-10-01 21:06:40 :  [path] /
python.x86_64                      2.7.5-69.el7_5                       @updates
ok  scenario succeeded
ok  text match /python.x86_64\s+(\d+)\.(\d+)\.(\d+)-/
ok  major version is more or equal 2
ok  minor version is more or equal 7
STATUS  SUCCEED
Enter fullscreen mode Exit fullscreen mode

Any other test case could be defined the same way. The approach looks like:

  1. Execute command(s)
  2. Check status code (optional)
  3. Validate commands output using check files

Check files contains static validation rules and pieces of code ( written on Perl, Bash, Ruby or Python ) to dynamically generate new rules.

Here more complicated example.

Monitoring a group of processes: 

$ nano story.bash

sleep 5 2>/dev/null 1>/dev/null &
sleep 5 2>/dev/null 1>/dev/null &
sleep 5 2>/dev/null 1>/dev/null &
sleep 5 2>/dev/null 1>/dev/null &
sleep 5 2>/dev/null 1>/dev/null &

ps aux | grep sleep | grep -v grep
Enter fullscreen mode Exit fullscreen mode 
$ nano story.check

begin:
sleep 5
sleep 5
sleep 5
sleep 5
sleep 5
end:
Enter fullscreen mode Exit fullscreen mode 
$ strun

2018-10-02 22:26:05 :  [path] /
root      48180  0.0  0.0   7772   664 pts/2    S+   22:26   0:00 sleep 5
root      48181  0.0  0.0   7772   748 pts/2    S+   22:26   0:00 sleep 5
root      48182  0.0  0.0   7772   640 pts/2    S+   22:26   0:00 sleep 5
root      48183  0.0  0.0   7772   712 pts/2    S+   22:26   0:00 sleep 5
root      48184  0.0  0.0   7772   748 pts/2    S+   22:26   0:00 sleep 5
ok      scenario succeeded
ok      [b] text has 'sleep 5'
ok      [b] text has 'sleep 5'
ok      [b] text has 'sleep 5'
ok      [b] text has 'sleep 5'
ok      [b] text has 'sleep 5'
STATUS  SUCCEED
Enter fullscreen mode Exit fullscreen mode

The same as above, but for N number of processes:

$ nano story.check

generator: [ "begin:", (map { "sleep 5" } (1 .. config()->{N})), "end:" ]

$ strun --param N=5
Enter fullscreen mode Exit fullscreen mode

Following examples of various checks happening in daily operations life, some of them are trivial, some are not, some are shamelessly "stolen" from the goss issues list to show that almost nothing is impossible with Outthentic check tool!

Check if an http response has a http header 

$ nano story.bash

curl -s -k -o /dev/null -D - http://headers.jsontest.com/ | grep Content-Type:

$ nano story.check

Content-Type: application/json
Enter fullscreen mode Exit fullscreen mode 
$ strun

2018-10-03 18:33:33 :  [path] /
Content-Type: application/json; charset=ISO-8859-1
ok      scenario succeeded
ok      text has 'Content-Type: application/json'
STATUS  SUCCEED
Enter fullscreen mode Exit fullscreen mode

Check if user running process 


$ nano story.bash

ps -o command -u root|grep /usr/lib/systemd/systemd-logind|grep -v grep


$ nano story.check

regexp: ^/usr/lib/systemd/systemd-logind$
Enter fullscreen mode Exit fullscreen mode 
$ strun

2018-10-03 18:50:24 :  [path] /
/usr/lib/systemd/systemd-logind
ok      scenario succeeded
ok      text match /^/usr/lib/systemd/systemd-logind$/
STATUS  SUCCEED
Enter fullscreen mode Exit fullscreen mode

Check if a kernel parameter greater than 

$ nano story.bash

sysctl -a 2>/dev/null | grep net.core.somaxconn


$ nano story.check

regexp: net.core.somaxconn = (\d+)

generator: <<CODE 
[ 
  "assert: ".( 
     capture()->[0] >= 128 ? 1 : 0 
   )." net.core.somaxconn is greater or equal then 128" 
]
CODE
Enter fullscreen mode Exit fullscreen mode 
$ strun

2018-10-03 19:52:01 :  [path] /
net.core.somaxconn = 128
ok      scenario succeeded
ok      text match /net.core.somaxconn = (\d+)/
ok      net.core.somaxconn is greater or equal then 128
STATUS  SUCCEED
Enter fullscreen mode Exit fullscreen mode

Basic auth for http/https checks 

$ nano story.bash

curl -u guest:guest -f -o /dev/null -s https://jigsaw.w3.org/HTTP/Basic/
Enter fullscreen mode Exit fullscreen mode 
$ strun

2018-10-03 21:53:38 :  [path] /
ok      scenario succeeded
STATUS  SUCCEED
Enter fullscreen mode Exit fullscreen mode

Check if a http port from a range is accessible 


$ nano story.bash

for i in 83 82 81 80; do
  curl http://sparrowhub.org:$i -o /dev/null -s -f --connect-timeout 3 && echo connected to $i
done


$ nano story.check

regexp: connected to \d+
Enter fullscreen mode Exit fullscreen mode 
$ strun

2018-10-03 22:06:03 :  [path] /
connected to 80
ok      scenario succeeded
ok      text match /connected to \d+/
STATUS  SUCCEED
Enter fullscreen mode Exit fullscreen mode

Check files by Glob/Regex names

Let's check that configuration files in /etc/ directory has root owner and 644 mode:


$ nano story.bash 

stat -c %U' '%a' '%n /etc/*.conf


$ nano story.check

regexp: (\S+)\s(\d\d\d)\s(\S+)

generator: <<CODE
  my @out;
  for my $f (@{captures()}){
    my $fname = $f->[2];
    push @out, "assert: ".( $f->[0] eq 'root' ? 1 : 0 )." $fname has a root owner";
    push @out, "assert: ".( $f->[1] eq '644' ? 1 : 0 )." $fname has a 644 mode";
  }
  [@out]
CODE
Enter fullscreen mode Exit fullscreen mode 

$ strun

2018-10-03 22:45:49 :  [path] /
root 644 /etc/dracut.conf
root 644 /etc/e2fsck.conf
root 644 /etc/host.conf
root 644 /etc/krb5.conf
root 644 /etc/ld.so.conf
root 640 /etc/libaudit.conf
root 644 /etc/libuser.conf
root 644 /etc/locale.conf
root 644 /etc/man_db.conf
root 644 /etc/mke2fs.conf
root 644 /etc/nsswitch.conf
root 644 /etc/resolv.conf
root 644 /etc/rsyncd.conf
root 644 /etc/sysctl.conf
root 644 /etc/vconsole.conf
root 644 /etc/yum.conf
ok      scenario succeeded
ok      text match /(\S+)\s(\d\d\d)\s(\S+)/
ok      /etc/dracut.conf has a root owner
ok      /etc/dracut.conf has a 644 mode
ok      /etc/e2fsck.conf has a root owner
ok      /etc/e2fsck.conf has a 644 mode
ok      /etc/host.conf has a root owner
ok      /etc/host.conf has a 644 mode
ok      /etc/krb5.conf has a root owner
ok      /etc/krb5.conf has a 644 mode
ok      /etc/ld.so.conf has a root owner
ok      /etc/ld.so.conf has a 644 mode
ok      /etc/libaudit.conf has a root owner
not ok  /etc/libaudit.conf has a 644 mode
ok      /etc/libuser.conf has a root owner
ok      /etc/libuser.conf has a 644 mode
ok      /etc/locale.conf has a root owner
ok      /etc/locale.conf has a 644 mode
ok      /etc/man_db.conf has a root owner
ok      /etc/man_db.conf has a 644 mode
ok      /etc/mke2fs.conf has a root owner
ok      /etc/mke2fs.conf has a 644 mode
ok      /etc/nsswitch.conf has a root owner
ok      /etc/nsswitch.conf has a 644 mode
ok      /etc/resolv.conf has a root owner
ok      /etc/resolv.conf has a 644 mode
ok      /etc/rsyncd.conf has a root owner
ok      /etc/rsyncd.conf has a 644 mode
ok      /etc/sysctl.conf has a root owner
ok      /etc/sysctl.conf has a 644 mode
ok      /etc/vconsole.conf has a root owner
ok      /etc/vconsole.conf has a 644 mode
ok      /etc/yum.conf has a root owner
ok      /etc/yum.conf has a 644 mode
STATUS  FAILED (2)
Enter fullscreen mode Exit fullscreen mode

On my docker machine /etc/libaudit.conf has a 640 mode which caused the test failure.

Check if a TCP address/port is being listened by a process

Let's validate that 22 port is listened by /usr/sbin/sshd

$ mkdir -p modules/file-proc

$ mkdir modules/netstat

$ nano modules/netstat/story.bash

netstat -nlpW|grep ':22'

$ nano modules/netstat/story.check

regexp: tcp\s+\d+\s+\d+\s+\S+?:22.*LISTEN\s+(\d+)

code: <<CODE
  our @pids;
  for my $f (@{captures()}){
    push @pids, $f->[0];
  }
CODE


$ nano modules/file-proc/story.bash

file /proc/$pid/exe

$ nano modules/file-proc/story.check

symbolic link to `/usr/sbin/sshd'


$ nano hook.pl

run_story "netstat";

for my $pid (our @pids){
  run_story "file-proc", { pid => $pid }
}

$ strun

2018-10-04 14:06:28 :  [path] /modules/netstat/
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      69/sshd
tcp6       0      0 :::22                   :::*                    LISTEN      69/sshd
ok      scenario succeeded
ok      text match /tcp\s+\d+\s+\d+\s+\S+?:22.*LISTEN\s+(\d+)/
2018-10-04 14:06:28 :  [path] /modules/file-proc/ [params] pid:69
/proc/69/exe: symbolic link to `/usr/sbin/sshd'
ok      scenario succeeded
ok      text has 'symbolic link to `/usr/sbin/sshd''
STATUS  SUCCEED
Enter fullscreen mode Exit fullscreen mode

Finally we can a whole bunch of tests we have written, recursively:

$ strun --recurse --format=production
2018-10-04 14:14:13 :  [path] /kernel-param/
2018-10-04 14:14:14 :  [path] /http-basic-auth/
2018-10-04 14:14:14 :  [path] /modules/netstat/
2018-10-04 14:14:14 :  [path] /modules/file-proc/ [params] pid:69
2018-10-04 14:14:14 :  [path] /http-header-response/
2018-10-04 14:14:15 :  [path] /package-exists/
2018-10-04 14:14:15 :  [path] /file-by-regexp/
not ok  /etc/libaudit.conf has a 644 mode
root 644 /etc/dracut.conf
root 644 /etc/e2fsck.conf
root 644 /etc/host.conf
root 644 /etc/krb5.conf
root 644 /etc/ld.so.conf
root 640 /etc/libaudit.conf
root 644 /etc/libuser.conf
root 644 /etc/locale.conf
root 644 /etc/man_db.conf
root 644 /etc/mke2fs.conf
root 644 /etc/nsswitch.conf
root 644 /etc/resolv.conf
root 644 /etc/rsyncd.conf
root 644 /etc/sysctl.conf
root 644 /etc/vconsole.conf
root 644 /etc/yum.conf
STATUS  FAILED (2)
2018-10-04 14:14:15 :  [path] /multiple-processes/
2018-10-04 14:14:16 :  [path] /process-run-user/
2018-10-04 14:14:16 :  [path] /range-of-ports/
STATUS  FAILED (254)
stories failed: /file-by-regexp
Enter fullscreen mode Exit fullscreen mode

We use production output format, which only shows details for failed tests (/file-by-regexp)

Tests reuse

We just have to package the test case we want to reuse into sparrow plugin:


$ cd kernel-param/

$ nano story.bash

param=$(config name)

sysctl  2>/dev/null -a | grep $param


$ nano story.check

generator: <<CODE

my $value = config()->{value}; 
my $param = config()->{name};

[ 
  'regexp: '.$param.' = (\d+)',

  "assert: ".( 
     capture()->[0] >= $value ? 1 : 0 
   )." $param is greater or equal then $value" 
]
CODE



$ nano sparrow.json
{

    "name": "kernel-param-check",
    "version": "0.0.1",
    "category" : "audit", 
    "description" : "check kernel parameter",
}

$ sparrow plg upload
Enter fullscreen mode Exit fullscreen mode

Then:

$ ssh host

$ sparrow index update

$ sparrow plg install kernel-param-check

$ sparrow plg run kernel-param-check \
--param name=net.core.somaxconn \
--param value=128
Enter fullscreen mode Exit fullscreen mode

Conclusion

Outthentic is flexible and efficient tool to get job done, when you need infrastructure tests. Rather then providing declarative DSL as tools like goss and inspec does it provides powerful tool-set to run arbitrary command and parse output. Sparrow plugin gives you a decent level of code reuse, when you can distribute independent small test sets across your environments.

It would be interesting to hear readers opinion on comparison this approach against existing ones ( goss, inspec ).

Top comments (0)

Read next

sk_rajibul_9ce58a68c43bb5 profile image

Simplifying Test Execution with pytest.main()

SK RAJIBUL -

hameed1206 profile image

TASK 14

Hameed Hussain -

bbylumi profile image

The Sweet Taste of Victory!

OLUWAPELUMI -

paulund profile image

How to debug axios requests

Paulund -