Ryan – Page 12 – Ryan Schulze

How to prevent changes to a tag via svn hook

February 4, 2015Ryanbash, hook, linux, subversion, svn

A colleague of mine recently asked if it was possible to keep people from committing changes to tags in subversion. I thought “Hey, that should be easy to do via the pre-commit hook. I bet someone already made one that I can just test and use“. Either my google-fu failed me or the request wasn’t as common as I had anticipated, because surprisingly I couldn’t find any hooks that truly accomplish blocking changes to a tag (probably right after I post this someone will say “hey, why didn’t you look $here, it is exactly what you wanted“).

I found people looking for such a feature, and I found a hook or two that kinda did what I needed (the best I could find was a hook that just blocked updates to /tags/* but it allowed deletes, adds and property changes), but none that really blocked all changes to tags. So I decided to just make my own configurable svn hook. You can tell it what to allow and what to block, and which directory to work on (since not everyone has the tags in their base directory of the repository).

You may have to change the SVNLOOK variable depending on where your svnlook binary is installed.

#!/bin/bash

#===============================================================================
#
#          FILE:  pre-commit
# 
#   DESCRIPTION:  pre-commit svn hook. configurable hook to block certain 
#                 actions on svn tags
#                 (e.g. blocking changes to existing tags)
# 
#  REQUIREMENTS:  svnlook
#        AUTHOR:  Ryan Schulze (rs), ryan@dopefish.de
#===============================================================================

#===============================================================================
# Path to tags directory in the repository
#===============================================================================
TAGS_PATH="^tags/"

#===============================================================================
# What to allow or block
#===============================================================================
UPDATE=block
DELETE=block
ADD=block
PROPERTIES=block

#===============================================================================

REPOS="${1}"
TXN="${2}"

SVNLOOK=/usr/bin/svnlook

ABORT=0

while read line
do
	# SVN Action for this element
	ACTION="${line%% *}"

	# Path for this element
	FILE_PATH="$(echo "${line#* }" | sed 's/^ *//')"

	# regex that skips entries for the top level directory of a specific tag (e.g. tags/1.5/)
	# so that we can still create and delete the tags themselves
	if [[ "${FILE_PATH}" =~ ${TAGS_PATH}[/]*[^/]+/.+$ ]] 
	then
		# A  - Item added to repository
		# D  - Item deleted from repository
		# U  - File contents changed
		# _U - Properties of item changed
		# UU - File contents and properties changed
		case ${ACTION} in 
			U | UU )  [[ "${UPDATE}" == 'block' ]] && ABORT=1 ;;
			D )       [[ "${DELETE}" == 'block' ]] && ABORT=1 ;;
			A )       [[ "${ADD}" == 'block' ]] && ABORT=1 ;;
			_U | UU ) [[ "${PROPERTIES}" == 'block' ]] && ABORT=1 ;;
		esac
	fi

	if [[ ${ABORT} -gt 0 ]]
	then
		echo "Cannot change tags!" 1>&2
		exit 1
	fi	

done < <(${SVNLOOK} changed -t "$TXN" "$REPOS")

# All checks passed, so allow the commit.
exit 0

#!/bin/bash

#===============================================================================

# FILE: pre-commit

# DESCRIPTION: pre-commit svn hook. configurable hook to block certain

# actions on svn tags

# (e.g. blocking changes to existing tags)

# REQUIREMENTS: svnlook

# AUTHOR: Ryan Schulze (rs), ryan@dopefish.de

#===============================================================================

# Path to tags directory in the repository

#===============================================================================

TAGS_PATH="^tags/"

#===============================================================================

# What to allow or block

#===============================================================================

UPDATE=block

DELETE=block

ADD=block

PROPERTIES=block

#===============================================================================

REPOS="${1}"

TXN="${2}"

SVNLOOK=/usr/bin/svnlook

ABORT=0

while read line

# SVN Action for this element

ACTION="${line%% *}"

# Path for this element

FILE_PATH="$(echo "${line#* }" | sed 's/^ *//')"

# regex that skips entries for the top level directory of a specific tag (e.g. tags/1.5/)

# so that we can still create and delete the tags themselves

if [[ "${FILE_PATH}" =~ ${TAGS_PATH}[/]*[^/]+/.+$ ]]

then

# A - Item added to repository

# D - Item deleted from repository

# U - File contents changed

# _U - Properties of item changed

# UU - File contents and properties changed

case ${ACTION} in

U | UU ) [[ "${UPDATE}" == 'block' ]] && ABORT=1 ;;

D ) [[ "${DELETE}" == 'block' ]] && ABORT=1 ;;

A ) [[ "${ADD}" == 'block' ]] && ABORT=1 ;;

_U | UU ) [[ "${PROPERTIES}" == 'block' ]] && ABORT=1 ;;

esac

if [[ ${ABORT} -gt 0 ]]

then

echo "Cannot change tags!" 1>&2

exit 1

done < <(${SVNLOOK} changed -t "$TXN" "$REPOS")

# All checks passed, so allow the commit.

exit 0

Native tcp/udp sockets in bash cheatsheet

December 9, 2014December 12, 2014Ryanbash, network

Bash has a nifty feature to open tcp and udp connections and read/write data to that connection. The Advanced Bash Scripting Guide and the bash man page offers some information and examples, and google has some odd examples, but all in all there isn’t much variety of information on the internet on the topic. This feature is enabled in Bash at compile time with the flag –enable-net-redirections

It works by assigning a connection to a file descriptor with exec. Protocol must be udp or tcp, hostname must be either an IP or a FQDN. Use any free file-descriptor (3 or higher usually).

exec {file-descriptor}<>/dev/{protocol}/{host}/{port}

1	exec {file-descriptor}<>/dev/{protocol}/{host}/{port}

Use &- to close the connection instead of leaving it in the CLOSE_WAIT status.

exec {file-descriptor}<>&-

1	exec {file-descriptor}<>&-

Basic Example:

exec 5<>/dev/tcp/www.ryanschulze.net/80
echo -e "GET / HTTP/1.1\n\n" >&5
cat <&5
exec 5<&-
exec 5>&-

exec 5<>/dev/tcp/www.ryanschulze.net/80

echo -e "GET / HTTP/1.1\n\n" >&5

cat <&5

exec 5<&-

exec 5>&-

You may have noticed that the cat will hang around a while after delivering the content. As long as the connection is established it will sit there and wait for data, which can be quite a while depending on the daemon on the other end. If you want to avoid having to wait, kill or ctrl-c the cat you can use read with an input timeout.

exec 5<>/dev/tcp/www.ryanschulze.net/80
echo -e "GET / HTTP/1.1\n\n" >&5
while read -t 2 line
do 
  echo $line
done <&5
exec 5<&-
exec 5>&-

exec 5<>/dev/tcp/www.ryanschulze.net/80

echo -e "GET / HTTP/1.1\n\n" >&5

while read -t 2 line

echo $line

done <&5

exec 5<&-

exec 5>&-

In the example if read has to wait longer than 2 seconds (-t 2) it will abort reading from the network connection.

If you only want to read a single line you can use head:

head -n1 <&5

1	head -n1 <&5

Although this will have the same timeout problems as cat if there is no more data but the connection is still established, it is useful and quick if you know exactly what kind of result you are expecting.

You don’t have to read the response right away, it will be buffered until you get around to accessing it (even if the other end terminates the connection).

What was and wasn’t fixed in bash after the Shellshock vulnerability (CVE-2014-6271)

September 27, 2014September 27, 2014Ryan

By now there are a few websites/blog posts online that explain CVE-2014-6271 (code being executed that was inserted after the function definition) and CVE-2014-7169 (parser error that also led to code being executed). Both CVEs have been patched. Check out “Quick notes about the bash bug, its impact, and the fixes so far“, “Shellshock proof of concept – Reverse shell“, or “Everything you need to know about the Shellshock Bash bug” if you want more information about the details and how CVE-2014-6271 worked.

Most websites focus on the remote exploitability of the vulnerability via CGI in web servers (understandably since this is the most dangerous aspect since how request headers are passed to scripts), and it didn’t take long after the announcement on the oss-security mailing list for requests starting to hit my webservers trying to exploit the vulnerability (from what I saw either people checking “how many servers does this really affect” or the more malicious “add your server to my DDoS botnet”).

What I would like to focus on is the functionality of exporting function definitions that has been drawn into the spotlight by CVE-2014-6271. You can define a function and make it accessible to the script/child shell that you invoke. This may sound really nifty (and it is … to a certain degree). The problem is that the script has no control over what functions are being imposed upon it. It is therefore possible to override any existing command or function, even builtin functions. No matter how hard you try to keep a sane environment within your script, anyone will be able to manipulate it from the outside (e.g. by overriding unset, set, …).

A small example script that initializes a variable, prints some output and then exits.

#!/bin/bash 

initialize() { 
	set -o nounset
	i=0
	trap cleanup TERM EXIT 
} 

cleanup() { 
	echo "exit"
	exit 0
} 

initialize

echo "Hello World"

if [[ ${i} -eq 10 ]]
then
	echo "should never be true"
fi

#!/bin/bash

initialize() {

set -o nounset

i=0

trap cleanup TERM EXIT

}

cleanup() {

echo "exit"

exit 0

}

initialize

echo "Hello World"

if [[ ${i} -eq 10 ]]

then

echo "should never be true"

If we execute the script we get the following output:

$ bash vulnrable.sh 
Hello World
exit

$ bash vulnrable.sh

Hello World

exit

Looks good, the if condition will never be true. Unless of course if we start overriding functions …

$ env trap='() { i=10; }' bash ./vulnrable.sh
Hello World
should never be true

$ env trap='() { i=10; }' bash ./vulnrable.sh

Hello World

should never be true

Sacrificed the trap function to set i=10. Or slightly more elaborate:

$ env echo='() { unset -f echo;echo ${@};i=10; }' bash ./vulnrable.sh
Hello World
should never be true
exit

$ env echo='() { unset -f echo;echo ${@};i=10; }' bash ./vulnrable.sh

Hello World

should never be true

exit

When echo is called we delete our function (so subsequent calls go to the original echo, to keep the code intact), execute an echo with whatever arguments were passed, and set i=10.

Obviously this is a simple example and you could override stuff like set, declare, test, …
Another problem is that this also affects binaries that execute a system() that calls a bash script. The following is a compiled simple C binary that uses system() to execute the bash script from the beginning of this posting.

$ env echo='() { unset -f echo;echo ${@};i=10; }' ./system_call
Hello World
should never be true
exit

$ env echo='() { unset -f echo;echo ${@};i=10; }' ./system_call

Hello World

should never be true

exit

As you see the environment gets passed along to the shell script. And in my opinion this is where it can start to get ugly. There are probably more bash scripts in your $PATH than you are aware of, do a quick
for dir in ${PATH//:/ }; do egrep "^#\!.*bash" ${dir}/* ;done and have a look for yourself.
If any of these scripts are called by a setuid/setgid binary that doesn’t sanitize and clean up the environment beforehand, you might have a serious problem on your hands.

Obviously if you have a program that is calling any kind of script, it is your job to make sure the environment is in a sane condition. At the same time I feel that having a more robust and secure way to implement exported functions would take away a lot of the pressure on the parent process ensuring sane conditions it may have itself inherited.

The topic is being extensively discussed by security experts and I expect the problem to be addressed shortly. In my opinion it should never be possible to override builtin functions externally (if a script itself wants to override a builtin, that’s fine with me, or have a switch to explicitly allow it). But since any solution will break the feature of exported functions in existing scripts it is a delicate problem to solve.

Downtime, Backups, and IPv6

February 16, 2014Ryanbackups, downtime, ipv6

Sorry for the unexpected downtime that lasted a few days and affected all the services here. One of the hard drives in the raid on my server suddenly showing miserable I/O performance, since the other drive was fine according to tests, SMART and logs I decided to take the slow drive offline to have it replaced. That turned out fatal since the other drive was also kaput and all my virtual machines on that server were now corrupted.

On the bright side the 2nd MX server worked fine and held all the email like it supposed to do until I got the main server back up and running (yaay to “exim -Mvb” to at least be able to read important mails that are in the mail server queue).

Long story short, since these drives have caused me much trouble in the past too, I decided to just get a new server with different hardware and restore everything from backups. Like any sysadmin I have my regular backups and scripts to check that they are working and occasionally test them to ensure I can restore a server from backups, but I’ve never had to really actually restore everything from a backup on one of my private servers. So when doing so I noticed minor things that aren’t perfect and needed to be changed to make my life easier in the future and decided to share my experience.

Backups. Instead of making one large archive with all the directories backed up (e.g. /etc, /home, /opt, …), split it up into multiple archives of the applications you are backing up (e.g. /etc/apache2, /var/www/). I should have known better since this is common practice at work. It makes your life a lot easier when restoring from a backup if you have one archive with everything in it associated to a specific application, Digging through a backup of /etc and trying to remember all sub-directories needed to get email back up and running was a chore (webinterface, greylisting, spamassassin, exim, dovecot … some of them store data you also want to restore in other parts of the filesystem). You can always tar all the application backups together into one package before uploading it to the backup server if you want.

Use some kind of configuration and/or deployment management software (Puppet, Chef, Ansible, Salt, own scripts, …). Don’t underestimate how relaxing it is to just press a button to get your server(s) back into the defined state you previously had, just drop in the data from the backups (configuration too depending on how you roll) and you are good to go. Getting networking up and running manually took me longer than it should have (VPN networks, routing on the VM host, sysctl settings, NAT, which interfaces are bridged, which are internal, guest network configuration, …), I’ve put that all into Ansible playbooks now so it’s just a press of a button.

IPv6 is still sorely underused in the internet. I always set it up on all my hosts and mainly I see it being used by core services like email servers, dns, package download servers of large distributions, sometimes ntp, that kind of stuff, not so much on normal websites. Ordering extra IPv4 IPs for my guests normally takes an extra day or so until they are assigned to me (and they each cost extra), so I do enjoy having a huge subnet ip IPv6 IPs free of charge to do with as I please. It’s nice to spin up a virtual server, assign it an IPv6 IP and have it online without having to worry about NAT or port forwarding or other stuff you are confronted with when getting a server/services online that reside on an internal IP.
While I like to push people to embrace IPv6 early and get used to it, it is also important to give it the same attention you give IPv4 to make sure that you aren’t opening yourself up to security problems. Make sure you have the same firewall policy for IPv6 as you do for IPv4 and IPv6 brings a few new features with it that IPv4 didn’t have. Unless you explicitly need/use them it is best to shut them off. Below I’ll ad some IPv6 settings you probably want to set per default unless you are explicitly using them.

net.ipv6.conf.all.autoconf=0
net.ipv6.conf.all.accept_dad=0
net.ipv6.conf.all.accept_ra=0
net.ipv6.conf.all.accept_ra_defrtr=0
net.ipv6.conf.all.accept_ra_rtr_pref=0
net.ipv6.conf.all.accept_ra_pinfo=0
net.ipv6.conf.all.accept_redirects=0
net.ipv6.conf.all.accept_source_route=0
net.ipv6.conf.default.autoconf=0
net.ipv6.conf.default.accept_dad=0
net.ipv6.conf.default.accept_ra=0
net.ipv6.conf.default.accept_ra_defrtr=0
net.ipv6.conf.default.accept_ra_rtr_pref=0
net.ipv6.conf.default.accept_ra_pinfo=0
net.ipv6.conf.default.accept_redirects=0
net.ipv6.conf.default.accept_source_route=0

net.ipv6.conf.all.autoconf=0

net.ipv6.conf.all.accept_dad=0

net.ipv6.conf.all.accept_ra=0

net.ipv6.conf.all.accept_ra_defrtr=0

net.ipv6.conf.all.accept_ra_rtr_pref=0

net.ipv6.conf.all.accept_ra_pinfo=0

net.ipv6.conf.all.accept_redirects=0

net.ipv6.conf.all.accept_source_route=0

net.ipv6.conf.default.autoconf=0

net.ipv6.conf.default.accept_dad=0

net.ipv6.conf.default.accept_ra=0

net.ipv6.conf.default.accept_ra_defrtr=0

net.ipv6.conf.default.accept_ra_rtr_pref=0

net.ipv6.conf.default.accept_ra_pinfo=0

net.ipv6.conf.default.accept_redirects=0

net.ipv6.conf.default.accept_source_route=0

How to install the latest Nmap for Debian/Ubuntu

August 21, 2013August 27, 2013Ryanbash, nmap, security

A quick & dirty script to download the latest version of nmap (sourcecode) and generate a deb and install it (so that it’s correctly in the package management). Yes, I know this is not much more than a glorified configure && make && checkinstall

#!/bin/bash

# some packages we will need. I put nmap in here to satisfy any dependencies like LUA
apt-get -qqy install checkinstall nmap build-essential

# get latest version
URL='http://nmap.org/dist/'
FILE="$(wget -qO - ${URL}|cut -d\" -f8|grep "tar.bz2"|sort -rn|head -n1)"

INSTALLED="$(dpkg -p nmap|grep Version:|awk '{print $2}')"

echo "Installed version: ${INSTALLED}, Downloaded version: ${FILE/.tar.bz2/}"
read -p "Compile and install new version? [Y/n] " answer
if [[ ${answer,,} = "n" ]] ; then
  exit 0
fi

echo "Downloading ${FILE}"
[[ -e  ${FILE} ]] || wget -qO ${FILE} ${URL}${FILE}
tar -xjf ${FILE}

cd ${FILE/.tar.bz2/}
./configure
if [[ ${FILE} =~ ^nmap-6.40. ]] ; then 
  # fixing an error in the 6.40 ncat Makefile
  sed -i 's/^LUA_LIBS.*/LUA_LIBS = $(top_srcdir)\/..\/liblua\/liblua.a -ldl -lm/1' ncat/Makefile
fi
make &amp;&amp; \
checkinstall -y

echo -e "\n Installed nmap version:"
$(which nmap) --version

#!/bin/bash

# some packages we will need. I put nmap in here to satisfy any dependencies like LUA

apt-get -qqy install checkinstall nmap build-essential

# get latest version

URL='http://nmap.org/dist/'

FILE="$(wget -qO - ${URL}|cut -d\" -f8|grep "tar.bz2"|sort -rn|head -n1)"

INSTALLED="$(dpkg -p nmap|grep Version:|awk '{print $2}')"

echo "Installed version: ${INSTALLED}, Downloaded version: ${FILE/.tar.bz2/}"

read -p "Compile and install new version? [Y/n] " answer

if [[ ${answer,,} = "n" ]] ; then

exit 0

echo "Downloading ${FILE}"

[[ -e ${FILE} ]] || wget -qO ${FILE} ${URL}${FILE}

tar -xjf ${FILE}

cd ${FILE/.tar.bz2/}

./configure

if [[ ${FILE} =~ ^nmap-6.40. ]] ; then

# fixing an error in the 6.40 ncat Makefile

sed -i 's/^LUA_LIBS.*/LUA_LIBS = $(top_srcdir)\/..\/liblua\/liblua.a -ldl -lm/1' ncat/Makefile

make && \

checkinstall -y

echo -e "\n Installed nmap version:"

$(which nmap) --version