Using regex comparision in bash and BASH_REMATCH

Bash supports regular expressions in comparisons via the =~ operator.  But what is rarely used or documented is that you can use the ${BASH_REMATCH[n]}  array to access successful matches (back-references to capture groups). So if you use parentheses for grouping ()  in your regex, you can access the content of that group.

Here is an example where I am parsing date placeholders in a text with an optional offset (e.g. |YYYY.MM.DD|+2 ). Storing the format and offset in separate groups:

 

 

 

Multiply floats by 10,100, … in bash

A short one today. Bash can only handle integer numbers and not floats, so when someone searches the internet on how to use math on floats in bash the solution they find is usually “use bc” and looks something like this:

Or if they want the result to be an integer:

It’s a fine solution, and readable (which can mean a lot for people maintaining scripts). But if all you want to do is multiply by 10,100,1000, … you can achieve this faster with a bit of string manipulation:

It just splits the number into two strings, and assembles it again with the decimal shifted. Have a look at substring_removal and substring_expansion for more examples on how to modify strings in bash. I’d highly suggest either sticking this in a separate function, or commenting the code since it isn’t necessarily obvious what is going on

Since it is all pure bash and doesn’t need to spawn external commands, it quicker (not that bc  is slow, but if you are doing a lot of calculations, it can add up). I know what you are thinking “if your goal is speed, you shouldn’t be using bash”, that doesn’t mean we can’t write efficient code.

API for Troy Hunts passwords list

TL;DR version: https://github.com/ryanschulze/password-check-api

So NIST updated their recommendations on passwords/authentication a few weeks ago. And while a lot of the reporting was about how password complexity was removed in favor of password length, one point I found intriguing was the suggestion to check if a users password falls into one of these categories:

  • Passwords obtained from previous breach corpuses.
  • Dictionary words.
  • Repetitive or sequential characters (e.g. ‘aaaaaa’, ‘1234abcd’).
  • Context-specific words, such as the name of the service, the username, and derivatives thereof.

Troy Hunt, the guy behind https://haveibeenpwned.com, deals with a lot of data breaches and made 320 Million passwords from breaches available (at the time of this posting) to help people with checking if passwords that were part of a data breach.

I threw together a small API that can make the data from Troy Hunt easily query-able (or any list of SHA1 hashes for that matter). This can be useful if you have multiple systems that want to query the data, or if you want the data on a separate system.

It’s nothing special, a MySQL backend, a Webserver and an API application using the Slim framework. It’s also stupid fast because there is nothing fancy or special about it. Since it uses a well documented framework it is also easy to change/extend/adjust to your specific requirements.

Running an OSSEC server with lots of agents

The default maximum amount of agents an OSSEC server supports is 256. That’s not very much. In order to increase that number, you need to recompile the source code. Instructions for that are in the official documentation

Depending on the defaults of your Linux Distribution, you may start noticing dropped UDP packets (depending on the amount of agents connecting and how much data they are sending to the server). An easy way to check is with  cat /proc/net/snmp | grep Udp\:|column -t  (check if RcvbufErrors or InErrors are increasing).

The following tweaks to the incoming network buffers resolved the situation for me, and can be a good starting point if you are having similar problems and looking at which settings to tweak. Focus on the rmem  buffers first.

 

Bash function to run commands against ansible hosts

I haven’t posted anything ansible related in a while, so here is a nifty little function I regularly use when I want to execute something on all (or a subset) of ansible hosts. It’s just a wrapper around ansible host -m script -a scriptname.sh  but adds –tree so that the output is stored and can easily be parsed by jq 

Usage example: