Programming – Page 3 – Ryan Schulze

How to check if a IP (ipv4) address is valid in pure Bash

July 25, 2012August 26, 2013Ryanbash, ipv4

Here is a small bash function to check if a IP is valid (4 octets, each octet < 256). I find it somewhat elegant since instead of using a lot of case/if/then constructs or a crazy long regex it splits the IP into each octet (and stores them in an array, and then uses a combination of regex and bit shifting to check each octet.

is_valid_ipv4() {
  local -a octets=( ${1//\./ } )
  local RETURNVALUE=0

  # return an error if the IP doesn't have exactly 4 octets
  [[ ${#octets[@]} -ne 4 ]] &amp;&amp; return 1

  for octet in ${octets[@]}
  do
    if [[ ${octet} =~ ^[0-9]{1,3}$ ]]
    then # shift number by 8 bits, anything larger than 255 will be > 0
      ((RETURNVALUE += octet>>8 ))
    else # octet wasn't numeric, return error
      return 1
    fi
  done
  return ${RETURNVALUE}
}

is_valid_ipv4() {

local -a octets=( ${1//\./ } )

local RETURNVALUE=0

# return an error if the IP doesn't have exactly 4 octets

[[ ${#octets[@]} -ne 4 ]] && return 1

for octet in ${octets[@]}

if [[ ${octet} =~ ^[0-9]{1,3}$ ]]

then # shift number by 8 bits, anything larger than 255 will be > 0

((RETURNVALUE += octet>>8 ))

else # octet wasn't numeric, return error

return 1

done

return ${RETURNVALUE}

}

The function will return 0 if the IP is valid, and 1 or higher if it encountered an error (you can check with the $? variable directly after calling the function)
Example:

is_valid_ipv4 ${ip}
if [[ $? -gt 0 ]]
then 
	echo "invalid IP"
else
	echo "valid IP"
fi

is_valid_ipv4 ${ip}

if [[ $? -gt 0 ]]

then

echo "invalid IP"

else

echo "valid IP"

How to add locking to a shell script (the easy way)

February 14, 2012February 18, 2012Ryanbash, flock

I haven’t posted anything with bash here for a while, so today I’ll throw in a little snippet to use flock to make sure a script is only running once. This is very handy in cron jobs that you want to run often, but there shouldn’t be multiple instances of the script running at the same time.
Since it is small and easy I’d recommend adding it to any code you don’t want running multiple times since “that script” you just wrote, that runs 10 minutes now, might turn into a monster in 6 months and run 45 minutes when things change (data grows, more stuff to do). Better safe than sorry.

Basically we rely on flock to do the heavy lifting and we just add some logic around it:

trap cleanup INT TERM EXIT # call cleanup() if script exits

cleanup() {
	  flock -u 200
}

exec 200&gt;/var/lock/my_script.lock

flock -xn 200 || exit 1 # exit if a lock exists

trap cleanup INT TERM EXIT # call cleanup() if script exits

cleanup() {

flock -u 200

}

exec 200>/var/lock/my_script.lock

flock -xn 200 || exit 1 # exit if a lock exists

man flock will show you more details to the parameters used and even some examples. Basically it will use trap to make sure the lock is released if the script ends in any way. 200 is a random file descriptor I chose for this example, it can be anything numeric. flock -xn means it will attempt to acquire an exclusive lock, and if that fails it will exit with an error.

Putting this somewhere at the top of your script will simply end the script if it finds an existing lock. flock has a few other options like -wait or nor using -n that allow you to not exit but wait for the lock to end (with wait a variable amount of seconds). And thus with a bit of creativity enabling you to only lock specific parts of the code (e.g. database calls, file changes, …) and handling failed lock attempts more gracefully than an exit.

How to increase Fraps performance with a ramdisk

November 10, 2011September 27, 2014Ryan2 Commentsbattlefield 3, fraps, IO, powershell, ramdisk, Windows

I recently started playing Battlefield 3 and remembered that I have a Fraps license so I installed it and started recording some stuff. Unsurprisingly the performance made a big dip when I recorded. A glance at my PC told me the harddrive was at fault, probably bringing the whole system down due to IO.

Since my PC has more than enough RAM I decided to set up a 5Gb Ramdisk to see if that helped. It did, when writing the video files to the ramdisk I hardly had any performance hit. Unfortunately 5GB isn’t going to last long while recording 1920×1080 @ 40FPS (a few minutes footage at most).

Here is my little cmd file to create a 5GB ramdisk as drive J: and format it for usage:

imdisk -d -m J:
imdisk -a -t vm -s 5G -o rw -m J:
format J: /A:64k /V:Ramdisk /FS:NTFS /Q

imdisk -d -m J:

imdisk -a -t vm -s 5G -o rw -m J:

format J: /A:64k /V:Ramdisk /FS:NTFS /Q

So my next thought was to see if I could write a script to move files off the ramdisk when they were done being written to by Fraps. This obviously was going to cause IO load … the reason we were having performance issues in the first place, so I was skeptical about if this was going to help any. Especially since I also had to move the files away quick enough so that the drive wouldn’t fill up completely with the next file Fraps was writing. I wrote a little powershell script for this (yeah, a *nix Sysadmin writing scripts in powershell …)

Here is my little powershell script to copy the finished files from my ramdisk to a normal HDD (please excuse possible ugliness, I’m a powershell noob):

$source = "j:\*.avi"
$target_dir = "f:\fraps\movies\"

while ($true) {
    $files = gci $source
    foreach ($file in $files){ 
        $srcfile = echo $file.fullname
        $dstfile = echo $file.basename
        if ($srcfile) {
            try {
                move-item "$srcfile" "$target_dir$dstfile.avi" -ea stop
                "$srcfile Moved"
            }
            catch {
                "$srcfile in use, cannot be copied right now"
            }
        }
    }
    sleep 5
}

$source = "j:\*.avi"

$target_dir = "f:\fraps\movies\"

while ($true) {

$files = gci $source

foreach ($file in $files){

$srcfile = echo $file.fullname

$dstfile = echo $file.basename

if ($srcfile) {

try {

move-item "$srcfile" "$target_dir$dstfile.avi" -ea stop

"$srcfile Moved"

}

catch {

"$srcfile in use, cannot be copied right now"

}

sleep 5

}

The last little problem I noticed is that the 5GB ramdrive wasn’t big enough (Fraps seems to create some dummy files and fills them up). Forcing Fraps to make smaller files by toggeling the recording fixed that though -> pressing F9 twice fast will drop a few frames though. I used my Logitech G13 for that, just had a key mapped to press F9 quickly every 60 seconds. The shortest gap I could get working reliably is 50ms.

This all probably sounds awfully complicated, but it works and solves my problem. Fraps is great software, but it would be immensly helpful if you could set the file size in the settings (instead of it defaulting to 4GB). Or, even better, if Fraps could rework their IO system to work more efficiently.

So to sum everything up:
– create ramdrive
– start script that copies files from the ramdrive to a normal HDD
– set fraps to store videos on the ramdrive
– start game, press F9 to start recording and then press the G13 key to toggle the F9 periodically

How to build an efficient GeoIP SQL table

August 22, 2011April 10, 2012Ryan1 Commentbash, GeoIP, MySQL

This here is a very handy little script I threw together to generate a geoip.sql table for quickly determining which country a IP is from. I already hear you saying “Just convert the IP to an INT and use BETWEEN, how hard can it be”. And you are right, that works. And it may even be your easiest solution, but it just isn’t fast. And if you are planning on hammering the table with thousands of queries you are going to end up looking for something fast.

A while back I found a very interesting posting at www.jcole.us that described how to use Spacial Indexes together with MySQL’s GIS to speed up the queries. The posting has been online for a while and both it and the replies are worth reading.

All I did was make a small bash script to download the current “lite” version of GeoIP CSV file from maxmind.com, use the information from the posting to throw/transform it into a local database table and dump out a .sql file that can be easily imported into any other database. The script isn’t failproof though, it expects your user to be able to use mysql and have permission to create databases/tables and “load data local infile”.

generate_geoip_sql.sh

#!/bin/bash
#===============================================================================
#
#          FILE: generate_geoip_sql.sh
#
#         USAGE: ./generate_geoip_sql.sh
#
#   DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com
#
#         USAGE: import sql into a database
#
#  REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE
#
#       EXAMPLE:
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))
#
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  INET_ATON('1.2.3.4') 
#                 BETWEEN ip_from AND ip_to;
#
#        AUTHOR:  Ryan Schulze (rs), ryan@dopefish.de
#
#===============================================================================

set -o nounset                              # Treat unset variables as an error

CSVFile="GeoIPCountryWhois.csv"

if [[ -e ${CSVFile} ]]
  then
  echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "
  read answer
  if [[ "$(echo $answer|tr "N" "n")" != "n" ]]
  then
    rm ${CSVFile}
  fi
fi

if [[ ! -e ${CSVFile} ]]
then
  tmpfile=$(mktemp)
  wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
  unzip ${tmpfile} &amp;&amp; rm ${tmpfile}
fi

echo -e "\ngenerating geoip.sql"

mysql -e "create database if not exists temp"
mysql -D temp -e "drop table if exists geo_ip"

echo '
CREATE TABLE geo_ip
(
  id           INT UNSIGNED  NOT NULL auto_increment,
  ip_poly      POLYGON       NOT NULL,
  ip_from      INT UNSIGNED  NOT NULL,
  ip_to        INT UNSIGNED  NOT NULL,
  country_code CHAR(2)       NOT NULL,
  country_name CHAR(50)      NOT NULL,
  PRIMARY KEY (id),
  SPATIAL INDEX (ip_poly)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"
INTO TABLE geo_ip
FIELDS
  TERMINATED BY ","
  ENCLOSED BY "\""
LINES
  TERMINATED BY "\n"
(
  @ip_from_string, @ip_to_string,
  @ip_from, @ip_to,
  @country_code, @country_string
)
SET
  id      := NULL,
  ip_from := @ip_from,
  ip_to   := @ip_to,
  ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(
    /* clockwise, 4 points and back to 0 */
    POINT(@ip_from, -1), /* 0, top left */
    POINT(@ip_to,   -1), /* 1, top right */
    POINT(@ip_to,    1), /* 2, bottom right */
    POINT(@ip_from,  1), /* 3, bottom left */
    POINT(@ip_from, -1)  /* 0, back to start */
  ))),
  country_code := @country_code,
  country_name := @country_string
;
' | mysql --local-infile=1 -D temp

mysqldump --opt temp geo_ip &gt; geoip.sql

echo -e "\ndone generating geoip.sql"

#!/bin/bash

#===============================================================================

# FILE: generate_geoip_sql.sh

# USAGE: ./generate_geoip_sql.sh

# DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com

# USAGE: import sql into a database

# REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE

# EXAMPLE:

# SELECT country_code,country_name

# FROM geo_ip

# WHERE MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))

# SELECT country_code,country_name

# FROM geo_ip

# WHERE INET_ATON('1.2.3.4')

# BETWEEN ip_from AND ip_to;

# AUTHOR: Ryan Schulze (rs), ryan@dopefish.de

#===============================================================================

set -o nounset # Treat unset variables as an error

CSVFile="GeoIPCountryWhois.csv"

if [[ -e ${CSVFile} ]]

then

echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "

read answer

if [[ "$(echo $answer|tr "N" "n")" != "n" ]]

then

rm ${CSVFile}

if [[ ! -e ${CSVFile} ]]

then

tmpfile=$(mktemp)

wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip

unzip ${tmpfile} && rm ${tmpfile}

echo -e "\ngenerating geoip.sql"

mysql -e "create database if not exists temp"

mysql -D temp -e "drop table if exists geo_ip"

echo '

CREATE TABLE geo_ip

(

id INT UNSIGNED NOT NULL auto_increment,

ip_poly POLYGON NOT NULL,

ip_from INT UNSIGNED NOT NULL,

ip_to INT UNSIGNED NOT NULL,

country_code CHAR(2) NOT NULL,

country_name CHAR(50) NOT NULL,

PRIMARY KEY (id),

SPATIAL INDEX (ip_poly)

) ENGINE=MyISAM DEFAULT CHARSET=latin1;

LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"

INTO TABLE geo_ip

FIELDS

TERMINATED BY ","

ENCLOSED BY "\""

LINES

TERMINATED BY "\n"

(

@ip_from_string, @ip_to_string,

@ip_from, @ip_to,

@country_code, @country_string

)

SET

id := NULL,

ip_from := @ip_from,

ip_to := @ip_to,

ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(

/* clockwise, 4 points and back to 0 */

POINT(@ip_from, -1), /* 0, top left */

POINT(@ip_to, -1), /* 1, top right */

POINT(@ip_to, 1), /* 2, bottom right */

POINT(@ip_from, 1), /* 3, bottom left */

POINT(@ip_from, -1) /* 0, back to start */

))),

country_code := @country_code,

country_name := @country_string

;

' | mysql --local-infile=1 -D temp

mysqldump --opt temp geo_ip > geoip.sql

echo -e "\ndone generating geoip.sql"

How to easily add colored text output in bash scripts

June 1, 2011September 27, 2014Ryan1 Commentbash, color, scripting

Here is small snippet that can give your shell scripts some nice output: colortext.sh As with the debug.sh script, just download it to the same directory as your own script and add it with

. colortext.sh

1	. colortext.sh

It contains one simple function called text with the syntax text “text to be output”. Color can be red, green, yellow, blue or grey. The function does not automatically add a linebreak to the putput, so pop a \n in there if you need it. I prefer using it together with printf for clean and easy color output.

Here are some examples of how the function can be used, and below the corresponding output:

. colortext.sh

echo "normal text"
text blue "blue text, "
text yellow "yellow text\n"

# Using it together with echo
echo -n "Status of script: [";text red "ERROR";echo "]"

# more elegant usage with printf
printf "Status of script: [%s]\n" $(text green "OK")

. colortext.sh

echo "normal text"

text blue "blue text, "

text yellow "yellow text\n"

# Using it together with echo

echo -n "Status of script: [";text red "ERROR";echo "]"

# more elegant usage with printf

printf "Status of script: [%s]\n" $(text green "OK")

Output:

normal text
blue text, yellow text
Status of script: [ERROR]
Status of script: [OK]