linux – Ryan Schulze

Playing around with eBPF

March 29, 2017July 17, 2017Ryankernel, linux

Here is a wonderful github repository for playing around with BPF https://github.com/iovisor/bcc

A script to diff files/directories on two different servers

February 6, 2016February 6, 2016Ryanbash, linux, scripting

Ok, short one today. This is a straightforward script that simplifies comparing directories on different servers. There is no magic in it, it just rsyncs the directories to a local temp directory and runs diff against them (then deletes the directory afterwards). Mainly intended for config files, I wouldn’t recommend trying to diff gigabytes of binaries with it.

#!/bin/bash 

set -o nounset

if [[ -z ${1:-} || -z ${2:-} || -z ${3:-} ]] ; then
	echo "Usage: ${0##*/} <file|directory> <server1> <server2>"
	echo
	echo "e.g. ./${0##*/} '/etc/pam.d/' 10.0.0.1 10.0.0.2"
	echo
	exit 0
fi

Dir="${1}"
IP1="${2}"
IP2="${3}"

tmpdir=$(mktemp -d)
trap 'rm -rf "${tmpdir}"' INT TERM EXIT

for ip in ${IP1} ${IP2}
do
	rsync -az "${ip}:${Dir}" "${tmpdir}/${ip}"
done
diff -BZEburN "${tmpdir}/${IP1}" "${tmpdir}/${IP2}" | colordiff --difftype=diffu

#!/bin/bash

set -o nounset

if [[ -z ${1:-} || -z ${2:-} || -z ${3:-} ]] ; then

echo "Usage: ${0##*/} <file|directory> <server1> <server2>"

echo

echo "e.g. ./${0##*/} '/etc/pam.d/' 10.0.0.1 10.0.0.2"

echo

exit 0

Dir="${1}"

IP1="${2}"

IP2="${3}"

tmpdir=$(mktemp -d)

trap 'rm -rf "${tmpdir}"' INT TERM EXIT

for ip in ${IP1} ${IP2}

rsync -az "${ip}:${Dir}" "${tmpdir}/${ip}"

done

diff -BZEburN "${tmpdir}/${IP1}" "${tmpdir}/${IP2}" | colordiff --difftype=diffu

Convert configuration files to ansible templates

April 8, 2015June 11, 2015Ryanansible, bash, linux, scripting

I’ve been playing around with ansible a lot lately, and I noticed that while changing stuff from “installed and configured manually” to “installed and configured by ansible” I was running into quite a few configuration files that needed to be manually turned into templates. It can be quite tedious to replace values in a configuration file with placeholders and put all those placeholders in a .yml file with default values.
Automating this is something I would have typically done in perl, but since I wanted to learn more about using regex in bash I decided to have a go at it in bash using regex and ${BASH_REMATCH}

The script takes a configuration file and spits out an ansible template, as well as the variable definitions you will need to add to your defaults/main.yml or vars/main.yml

The whole script is a bit to long to post here, but the interesting part is:

if [[ ${line} =~ ^([^#][^\ ]+)[\ ]*[${Separator}][\ ]*([^\ ]+)$ ]] ; then
	VariableName="${Prefix}_${BASH_REMATCH[1]//-/_}" # create a name for this configuration variable
	VariableName="${VariableName,,}" # make lowercase
	sed -ri "s/^(${BASH_REMATCH[1]}[\ ]*[${Separator}][\ ]*).+$/\1{{ ${VariableName} }}/" "${Template}" # change the ansible template
	printf "%-40s %s\n" "${VariableName}:" "'${BASH_REMATCH[2]}'" # print variable info to stdout 
fi

if [[ ${line} =~ ^([^#][^\ ]+)[\ ]*[${Separator}][\ ]*([^\ ]+)$ ]] ; then

VariableName="${Prefix}_${BASH_REMATCH[1]//-/_}" # create a name for this configuration variable

VariableName="${VariableName,,}" # make lowercase

sed -ri "s/^(${BASH_REMATCH[1]}[\ ]*[${Separator}][\ ]*).+$/\1{{ ${VariableName} }}/" "${Template}" # change the ansible template

printf "%-40s %s\n" "${VariableName}:" "'${BASH_REMATCH[2]}'" # print variable info to stdout

(You can download the full script here ansible_template.sh).

You can use regular expressions in a [[ ]] with =~ (e.g. if [[ “boot” =~ ^b ]]), and you can access the result of the regular expression by using ( ) to mark what parts of the result to store and access them via $BASH_REMATCH (comparable to how you would do it for other languages). Here I am parsing out anything that looks like a key=value from the configfile (with multiple possible separators) and storing the results in BASH_REMATCH[1] and BASH_REMATCH[2]

Usage of the script is pretty straightforward. you give it a prefix for the variable names (so you don’t end up with multiple roles all using a common variable name like “port”), and either a local or remote file to work with, and it spits out something like this:

$ ./ansible_template.sh php webserver.somewhere.tld:/etc/php5/conf.d/xcache.ini

- name: Template

template: src={{ item.local }} dest={{ item.remote }} owner={{ item.owner }} group={{ item.group }} mode={{ item.mode }}

with_items:

- { local: 'xcache.ini.j2', remote: '/etc/php5/conf.d/xcache.ini', owner: 'root', group: 'root', mode: '0644' }

php_zend_extension: '/usr/lib/php5/20090626/xcache.so'

php_xcache.admin.enable_auth: 'On'

php_xcache.admin.user: 'admin'

php_xcache.admin.pass: 'ea6299af10b40ba80236a0f015ed627d'

php_xcache.shm_scheme: 'mmap'

php_xcache.size: '16M'

php_xcache.count: '1'

php_xcache.slots: '8K'

php_xcache.ttl: '0'

There a tons of different configuration file formats out there so this script won’t work perfectly 100% of the time, but it does do quite well and reduces the manually copy&pasting to a minimum.

$ cat xcache.ini.j2

; configuration for php Xcache module

[xcache-common]

zend_extension = {{ php_zend_extension }}

[xcache.admin]

xcache.admin.enable_auth = {{ php_xcache.admin.enable_auth }}

xcache.admin.user = "{{ php_xcache.admin.user }}"

xcache.admin.pass = "{{ php_xcache.admin.pass }}"

[xcache]

xcache.shm_scheme = "{{ php_xcache.shm_scheme }}"

xcache.size = {{ php_xcache.size }}

xcache.count = {{ php_xcache.count }}

xcache.slots = {{ php_xcache.slots }}

xcache.ttl = {{ php_xcache.ttl }}

...

How to prevent changes to a tag via svn hook

February 4, 2015Ryanbash, hook, linux, subversion, svn

A colleague of mine recently asked if it was possible to keep people from committing changes to tags in subversion. I thought “Hey, that should be easy to do via the pre-commit hook. I bet someone already made one that I can just test and use“. Either my google-fu failed me or the request wasn’t as common as I had anticipated, because surprisingly I couldn’t find any hooks that truly accomplish blocking changes to a tag (probably right after I post this someone will say “hey, why didn’t you look $here, it is exactly what you wanted“).

I found people looking for such a feature, and I found a hook or two that kinda did what I needed (the best I could find was a hook that just blocked updates to /tags/* but it allowed deletes, adds and property changes), but none that really blocked all changes to tags. So I decided to just make my own configurable svn hook. You can tell it what to allow and what to block, and which directory to work on (since not everyone has the tags in their base directory of the repository).

You may have to change the SVNLOOK variable depending on where your svnlook binary is installed.

#!/bin/bash

#===============================================================================
#
#          FILE:  pre-commit
# 
#   DESCRIPTION:  pre-commit svn hook. configurable hook to block certain 
#                 actions on svn tags
#                 (e.g. blocking changes to existing tags)
# 
#  REQUIREMENTS:  svnlook
#        AUTHOR:  Ryan Schulze (rs), ryan@dopefish.de
#===============================================================================

#===============================================================================
# Path to tags directory in the repository
#===============================================================================
TAGS_PATH="^tags/"

#===============================================================================
# What to allow or block
#===============================================================================
UPDATE=block
DELETE=block
ADD=block
PROPERTIES=block

#===============================================================================

REPOS="${1}"
TXN="${2}"

SVNLOOK=/usr/bin/svnlook

ABORT=0

while read line
do
	# SVN Action for this element
	ACTION="${line%% *}"

	# Path for this element
	FILE_PATH="$(echo "${line#* }" | sed 's/^ *//')"

	# regex that skips entries for the top level directory of a specific tag (e.g. tags/1.5/)
	# so that we can still create and delete the tags themselves
	if [[ "${FILE_PATH}" =~ ${TAGS_PATH}[/]*[^/]+/.+$ ]] 
	then
		# A  - Item added to repository
		# D  - Item deleted from repository
		# U  - File contents changed
		# _U - Properties of item changed
		# UU - File contents and properties changed
		case ${ACTION} in 
			U | UU )  [[ "${UPDATE}" == 'block' ]] && ABORT=1 ;;
			D )       [[ "${DELETE}" == 'block' ]] && ABORT=1 ;;
			A )       [[ "${ADD}" == 'block' ]] && ABORT=1 ;;
			_U | UU ) [[ "${PROPERTIES}" == 'block' ]] && ABORT=1 ;;
		esac
	fi

	if [[ ${ABORT} -gt 0 ]]
	then
		echo "Cannot change tags!" 1>&2
		exit 1
	fi	

done < <(${SVNLOOK} changed -t "$TXN" "$REPOS")

# All checks passed, so allow the commit.
exit 0

#!/bin/bash

#===============================================================================

# FILE: pre-commit

# DESCRIPTION: pre-commit svn hook. configurable hook to block certain

# actions on svn tags

# (e.g. blocking changes to existing tags)

# REQUIREMENTS: svnlook

# AUTHOR: Ryan Schulze (rs), ryan@dopefish.de

#===============================================================================

# Path to tags directory in the repository

#===============================================================================

TAGS_PATH="^tags/"

#===============================================================================

# What to allow or block

#===============================================================================

UPDATE=block

DELETE=block

ADD=block

PROPERTIES=block

#===============================================================================

REPOS="${1}"

TXN="${2}"

SVNLOOK=/usr/bin/svnlook

ABORT=0

while read line

# SVN Action for this element

ACTION="${line%% *}"

# Path for this element

FILE_PATH="$(echo "${line#* }" | sed 's/^ *//')"

# regex that skips entries for the top level directory of a specific tag (e.g. tags/1.5/)

# so that we can still create and delete the tags themselves

if [[ "${FILE_PATH}" =~ ${TAGS_PATH}[/]*[^/]+/.+$ ]]

then

# A - Item added to repository

# D - Item deleted from repository

# U - File contents changed

# _U - Properties of item changed

# UU - File contents and properties changed

case ${ACTION} in

U | UU ) [[ "${UPDATE}" == 'block' ]] && ABORT=1 ;;

D ) [[ "${DELETE}" == 'block' ]] && ABORT=1 ;;

A ) [[ "${ADD}" == 'block' ]] && ABORT=1 ;;

_U | UU ) [[ "${PROPERTIES}" == 'block' ]] && ABORT=1 ;;

esac

if [[ ${ABORT} -gt 0 ]]

then

echo "Cannot change tags!" 1>&2

exit 1

done < <(${SVNLOOK} changed -t "$TXN" "$REPOS")

# All checks passed, so allow the commit.

exit 0

Playing around with DD-WRT

December 24, 2011Ryanaccess point, asus, dd-wrt, IDS, linux, security, wifi, wlan

I’m currently playing around with my two WL-330GE Access points from asus (see an older posting). Since that posting I was a bit creative using the existing ethernet cabling and ports in the apartment to be able to retire the WiFi bridge without having any cable going through the apartment.

So I decided to use the two access points for something more useful. I’m playing around with dd-wrt to build configurations to use them as WiFi probes (for an IDS), or as Rouge Access Points (for demonstration purposes and to test wireless IDS solutions). I might compile my own dd-wrt version for the rouge version, there are a few things I miss to build a truly evil device.

I like the size of the devices (very compact) and that you can power them with 5V (you can run them off any USB port, right now the one here is hooked up to the USB port of a printer intended for cameras) the only thing missing to make them perfect would be Power-over-Ethernet and maybe a GSM interface to upload data online.

Fun having a cheap and small device like this with Wifi and ethernet running linux. Provides lots of possibilities and fun.