Tag: bash

A script to diff files/directories on two different servers

Ryanbash, linux, scripting

Ok,  short one today. This is a straightforward script that simplifies comparing directories on different servers. There is no magic in it, it just rsyncs the directories to a local temp directory and runs diff against them (then deletes the directory afterwards). Mainly intended for config files, I wouldn’t recommend trying to diff gigabytes of binaries with it.

 

Renewing “Let’s Encrypt” SSL certificates

Ryanbash, ssl

Let’s Encrypt provides free DV SSL certificates for everyone and is now in the open beta phase. I’m not going to go into the details of which of the clients are best, since that depends entirely on your use case (I use acme-tiny and a rule in varnish to intercept all calls to /.well-known/acme-challenge/).

Since the certificates are only valid for 90 days, I often see people suggesting to just renew them via cronjob every 2 months. I find this to be really awful advice, if that renewal fails for any reasons (network problems, local problems, problems with let’s encrypt) the next renewal is a month after the certificate expired. It is also pretty inflexible (what if you would rather prefer to renew them after 80 days).

I use openssl to check daily how long the certificate is still valid, and if a threshold has been reached it tries to renew the certificate (I believe the official client has this functionality too). And if the certificate isn’t renewed by a 2nd threshold, it sends an email altering the admin of the problem (for manually intervening and fixing whatever went wrong).

At the end of this posting I’ll add the complete script, but the quickest way to check how long a certificate is still valid is to use openssl x509 -in -checkend. It will return 0 if the file is still valid in x seconds, and 1 if the certificate doesn’t exist or if the certificate will be expired by then. Just multiply the number of days by 86400 and check if the certificate is still valid:

The openssl binary has a few nice options for looking at certificates (both local files and remotely connecting to a server and looking at the provided certificate)
Show information about a local certificate file: openssl x509 -text -noout -in
Connect to a remote server and display the certificate provided: openssl s_client -showcerts -servername foo.bar -connect IP:PORT | openssl x509 -text -noout  (–servername foo.bar is only required if you are connecting to a server and need to use SNI to request a cert for a specific domain, i.e. a webserver providing multiple domains on port 443 via SNI. It can of course be omitted if you don’t need it.)

This is the full script I use for checking and renewing certs. It basically just loops through a list of domains, checks if any of the date thresholds are met and then renews certificates/send emails.

Convert configuration files to ansible templates

Ryanansible, bash, linux, scripting

I’ve been playing around with ansible a lot lately, and I noticed that while changing stuff from “installed and configured manually” to “installed and configured by ansible” I was running into quite a few configuration files that needed to be manually turned into templates. It can be quite tedious to replace values in a configuration file with placeholders and put all those placeholders in a .yml file with default values.
Automating this is something I would have typically done in perl, but since I wanted to learn more about using regex in bash I decided to have a go at it in bash using regex and ${BASH_REMATCH}

The script takes a configuration file and spits out an ansible template, as well as the variable definitions you will need to add to your defaults/main.yml or vars/main.yml

The whole script is a bit to long to post here, but the interesting part is:

(You can download the full script here ansible_template.sh).

You can use regular expressions in a [[ ]] with =~ (e.g. if [[ “boot” =~ ^b ]]), and you can access the result of the regular expression by using ( ) to mark what parts of the result to store and access them via $BASH_REMATCH (comparable to how you would do it for other languages). Here I am parsing out anything that looks like a key=value from the configfile (with multiple possible separators) and storing the results in BASH_REMATCH[1] and BASH_REMATCH[2]

Usage of the script is pretty straightforward. you give it a prefix for the variable names (so you don’t end up with multiple roles all using a common variable name like “port”), and either a local or remote file to work with, and it spits out something like this:

There a tons of different configuration file formats out there so this script won’t work perfectly 100% of the time, but it does do quite well and reduces the manually copy&pasting to a minimum.

How to prevent changes to a tag via svn hook

Ryanbash, hook, linux, subversion, svn

A colleague of mine recently asked if it was possible to keep people from committing changes to tags in subversion. I thought “Hey, that should be easy to do via the pre-commit hook. I bet someone already made one that I can just test and use“. Either my google-fu failed me or the request wasn’t as common as I had anticipated, because surprisingly I couldn’t find any hooks that truly accomplish blocking changes to a tag (probably right after I post this someone will say “hey, why didn’t you look $here, it is exactly what you wanted“).

I found people looking for such a feature, and I found a hook or two that kinda did what I needed (the best I could find was a hook that just blocked updates to /tags/* but it allowed deletes, adds and property changes), but none that really blocked all changes to tags. So I decided to just make my own configurable svn hook. You can tell it what to allow and what to block, and which directory to work on (since not everyone has the tags in their base directory of the repository).

You may have to change the SVNLOOK variable depending on where your svnlook binary is installed.

 

Native tcp/udp sockets in bash cheatsheet

Ryanbash, network

Bash has a nifty feature to open tcp and udp connections and read/write data to that connection. The Advanced Bash Scripting Guide and the bash man page offers some information and examples, and google has some odd examples, but all in all there isn’t much variety of information on the internet on the topic. This feature is enabled in Bash at compile time with the flag –enable-net-redirections

It works by assigning a connection to a file descriptor with exec. Protocol must be udp or tcp, hostname must be either an IP or a FQDN. Use any free file-descriptor (3 or higher usually).

Use &- to close the connection instead of leaving it in the CLOSE_WAIT status.

Basic Example:

You may have noticed that the cat will hang around a while after delivering the content. As long as the connection is established it will sit there and wait for data, which can be quite a while depending on the daemon on the other end. If you want to avoid having to wait, kill or ctrl-c the cat  you can use read with an input timeout.

In the example if read has to wait longer than 2 seconds (-t 2) it will abort reading from the network connection.

If you only want to read a single line you can use head:

Although this will have the same timeout problems as cat if there is no more data but the connection is still established, it is useful and quick if you know exactly what kind of result you are expecting.

You don’t have to read the response right away, it will be buffered until you get around to accessing it (even if the other end terminates the connection).