Tech – Page 11 – Ryan Schulze

New Server (with IPv6)

February 18, 2012May 13, 2013Ryanhetzner, ipv6

I got a new server and I finished moving everything to the new hardware this week. If you notice anything behaving strange just drop me a note. I also set up IPv6 on all my VMs so will be playing around with that a bit. It’s a shame that hardly any providers give customers IPv6 addresses, migrating from IPv4 to IPv6 will take forever if none of the users can use them (and in the meantime IPv4 addresses are getting more and more expensive for me).

The new server is from hetzner since I have been very satisfied with them in the past. Impressive hardware features for low prices and (what really makes me keep choosing them) fantastic technical support. I’ve been with a few companies in the past, but none have had as good support as hetzner. Sooner or later hardware will die, and having a hosting provider that responds promptly and knowledgeably to request is priceless.

How to add locking to a shell script (the easy way)

February 14, 2012February 18, 2012Ryanbash, flock

I haven’t posted anything with bash here for a while, so today I’ll throw in a little snippet to use flock to make sure a script is only running once. This is very handy in cron jobs that you want to run often, but there shouldn’t be multiple instances of the script running at the same time.
Since it is small and easy I’d recommend adding it to any code you don’t want running multiple times since “that script” you just wrote, that runs 10 minutes now, might turn into a monster in 6 months and run 45 minutes when things change (data grows, more stuff to do). Better safe than sorry.

Basically we rely on flock to do the heavy lifting and we just add some logic around it:

trap cleanup INT TERM EXIT # call cleanup() if script exits

cleanup() {
	  flock -u 200
}

exec 200&gt;/var/lock/my_script.lock

flock -xn 200 || exit 1 # exit if a lock exists

trap cleanup INT TERM EXIT # call cleanup() if script exits

cleanup() {

flock -u 200

}

exec 200>/var/lock/my_script.lock

flock -xn 200 || exit 1 # exit if a lock exists

man flock will show you more details to the parameters used and even some examples. Basically it will use trap to make sure the lock is released if the script ends in any way. 200 is a random file descriptor I chose for this example, it can be anything numeric. flock -xn means it will attempt to acquire an exclusive lock, and if that fails it will exit with an error.

Putting this somewhere at the top of your script will simply end the script if it finds an existing lock. flock has a few other options like -wait or nor using -n that allow you to not exit but wait for the lock to end (with wait a variable amount of seconds). And thus with a bit of creativity enabling you to only lock specific parts of the code (e.g. database calls, file changes, …) and handling failed lock attempts more gracefully than an exit.

How to increase Fraps performance with a ramdisk

November 10, 2011September 27, 2014Ryan2 Commentsbattlefield 3, fraps, IO, powershell, ramdisk, Windows

I recently started playing Battlefield 3 and remembered that I have a Fraps license so I installed it and started recording some stuff. Unsurprisingly the performance made a big dip when I recorded. A glance at my PC told me the harddrive was at fault, probably bringing the whole system down due to IO.

Since my PC has more than enough RAM I decided to set up a 5Gb Ramdisk to see if that helped. It did, when writing the video files to the ramdisk I hardly had any performance hit. Unfortunately 5GB isn’t going to last long while recording 1920×1080 @ 40FPS (a few minutes footage at most).

Here is my little cmd file to create a 5GB ramdisk as drive J: and format it for usage:

imdisk -d -m J:
imdisk -a -t vm -s 5G -o rw -m J:
format J: /A:64k /V:Ramdisk /FS:NTFS /Q

imdisk -d -m J:

imdisk -a -t vm -s 5G -o rw -m J:

format J: /A:64k /V:Ramdisk /FS:NTFS /Q

So my next thought was to see if I could write a script to move files off the ramdisk when they were done being written to by Fraps. This obviously was going to cause IO load … the reason we were having performance issues in the first place, so I was skeptical about if this was going to help any. Especially since I also had to move the files away quick enough so that the drive wouldn’t fill up completely with the next file Fraps was writing. I wrote a little powershell script for this (yeah, a *nix Sysadmin writing scripts in powershell …)

Here is my little powershell script to copy the finished files from my ramdisk to a normal HDD (please excuse possible ugliness, I’m a powershell noob):

$source = "j:\*.avi"
$target_dir = "f:\fraps\movies\"

while ($true) {
    $files = gci $source
    foreach ($file in $files){ 
        $srcfile = echo $file.fullname
        $dstfile = echo $file.basename
        if ($srcfile) {
            try {
                move-item "$srcfile" "$target_dir$dstfile.avi" -ea stop
                "$srcfile Moved"
            }
            catch {
                "$srcfile in use, cannot be copied right now"
            }
        }
    }
    sleep 5
}

$source = "j:\*.avi"

$target_dir = "f:\fraps\movies\"

while ($true) {

$files = gci $source

foreach ($file in $files){

$srcfile = echo $file.fullname

$dstfile = echo $file.basename

if ($srcfile) {

try {

move-item "$srcfile" "$target_dir$dstfile.avi" -ea stop

"$srcfile Moved"

}

catch {

"$srcfile in use, cannot be copied right now"

}

sleep 5

}

The last little problem I noticed is that the 5GB ramdrive wasn’t big enough (Fraps seems to create some dummy files and fills them up). Forcing Fraps to make smaller files by toggeling the recording fixed that though -> pressing F9 twice fast will drop a few frames though. I used my Logitech G13 for that, just had a key mapped to press F9 quickly every 60 seconds. The shortest gap I could get working reliably is 50ms.

This all probably sounds awfully complicated, but it works and solves my problem. Fraps is great software, but it would be immensly helpful if you could set the file size in the settings (instead of it defaulting to 4GB). Or, even better, if Fraps could rework their IO system to work more efficiently.

So to sum everything up:
– create ramdrive
– start script that copies files from the ramdrive to a normal HDD
– set fraps to store videos on the ramdrive
– start game, press F9 to start recording and then press the G13 key to toggle the F9 periodically

How to build an efficient GeoIP SQL table

August 22, 2011April 10, 2012Ryan1 Commentbash, GeoIP, MySQL

This here is a very handy little script I threw together to generate a geoip.sql table for quickly determining which country a IP is from. I already hear you saying “Just convert the IP to an INT and use BETWEEN, how hard can it be”. And you are right, that works. And it may even be your easiest solution, but it just isn’t fast. And if you are planning on hammering the table with thousands of queries you are going to end up looking for something fast.

A while back I found a very interesting posting at www.jcole.us that described how to use Spacial Indexes together with MySQL’s GIS to speed up the queries. The posting has been online for a while and both it and the replies are worth reading.

All I did was make a small bash script to download the current “lite” version of GeoIP CSV file from maxmind.com, use the information from the posting to throw/transform it into a local database table and dump out a .sql file that can be easily imported into any other database. The script isn’t failproof though, it expects your user to be able to use mysql and have permission to create databases/tables and “load data local infile”.

generate_geoip_sql.sh

#!/bin/bash
#===============================================================================
#
#          FILE: generate_geoip_sql.sh
#
#         USAGE: ./generate_geoip_sql.sh
#
#   DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com
#
#         USAGE: import sql into a database
#
#  REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE
#
#       EXAMPLE:
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))
#
#                 SELECT country_code,country_name
#                 FROM   geo_ip
#                 WHERE  INET_ATON('1.2.3.4') 
#                 BETWEEN ip_from AND ip_to;
#
#        AUTHOR:  Ryan Schulze (rs), ryan@dopefish.de
#
#===============================================================================

set -o nounset                              # Treat unset variables as an error

CSVFile="GeoIPCountryWhois.csv"

if [[ -e ${CSVFile} ]]
  then
  echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "
  read answer
  if [[ "$(echo $answer|tr "N" "n")" != "n" ]]
  then
    rm ${CSVFile}
  fi
fi

if [[ ! -e ${CSVFile} ]]
then
  tmpfile=$(mktemp)
  wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip
  unzip ${tmpfile} &amp;&amp; rm ${tmpfile}
fi

echo -e "\ngenerating geoip.sql"

mysql -e "create database if not exists temp"
mysql -D temp -e "drop table if exists geo_ip"

echo '
CREATE TABLE geo_ip
(
  id           INT UNSIGNED  NOT NULL auto_increment,
  ip_poly      POLYGON       NOT NULL,
  ip_from      INT UNSIGNED  NOT NULL,
  ip_to        INT UNSIGNED  NOT NULL,
  country_code CHAR(2)       NOT NULL,
  country_name CHAR(50)      NOT NULL,
  PRIMARY KEY (id),
  SPATIAL INDEX (ip_poly)
) ENGINE=MyISAM DEFAULT CHARSET=latin1;
LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"
INTO TABLE geo_ip
FIELDS
  TERMINATED BY ","
  ENCLOSED BY "\""
LINES
  TERMINATED BY "\n"
(
  @ip_from_string, @ip_to_string,
  @ip_from, @ip_to,
  @country_code, @country_string
)
SET
  id      := NULL,
  ip_from := @ip_from,
  ip_to   := @ip_to,
  ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(
    /* clockwise, 4 points and back to 0 */
    POINT(@ip_from, -1), /* 0, top left */
    POINT(@ip_to,   -1), /* 1, top right */
    POINT(@ip_to,    1), /* 2, bottom right */
    POINT(@ip_from,  1), /* 3, bottom left */
    POINT(@ip_from, -1)  /* 0, back to start */
  ))),
  country_code := @country_code,
  country_name := @country_string
;
' | mysql --local-infile=1 -D temp

mysqldump --opt temp geo_ip &gt; geoip.sql

echo -e "\ndone generating geoip.sql"

#!/bin/bash

#===============================================================================

# FILE: generate_geoip_sql.sh

# USAGE: ./generate_geoip_sql.sh

# DESCRIPTION: Generates a optimized sql dump with geoipdata from maxmind.com

# USAGE: import sql into a database

# REQUIREMENTS: mysql database rights to create databases and LOAD DATA LOCAL INFILE

# EXAMPLE:

# SELECT country_code,country_name

# FROM geo_ip

# WHERE MBRCONTAINS(ip_poly, POINTFROMWKB(POINT(INET_ATON('1.2.3.4'), 0)))

# SELECT country_code,country_name

# FROM geo_ip

# WHERE INET_ATON('1.2.3.4')

# BETWEEN ip_from AND ip_to;

# AUTHOR: Ryan Schulze (rs), ryan@dopefish.de

#===============================================================================

set -o nounset # Treat unset variables as an error

CSVFile="GeoIPCountryWhois.csv"

if [[ -e ${CSVFile} ]]

then

echo -en "\n${CSVFile} already exists, download a newer version? [Y/n]: "

read answer

if [[ "$(echo $answer|tr "N" "n")" != "n" ]]

then

rm ${CSVFile}

if [[ ! -e ${CSVFile} ]]

then

tmpfile=$(mktemp)

wget -O ${tmpfile} http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip

unzip ${tmpfile} && rm ${tmpfile}

echo -e "\ngenerating geoip.sql"

mysql -e "create database if not exists temp"

mysql -D temp -e "drop table if exists geo_ip"

echo '

CREATE TABLE geo_ip

(

id INT UNSIGNED NOT NULL auto_increment,

ip_poly POLYGON NOT NULL,

ip_from INT UNSIGNED NOT NULL,

ip_to INT UNSIGNED NOT NULL,

country_code CHAR(2) NOT NULL,

country_name CHAR(50) NOT NULL,

PRIMARY KEY (id),

SPATIAL INDEX (ip_poly)

) ENGINE=MyISAM DEFAULT CHARSET=latin1;

LOAD DATA LOCAL INFILE "GeoIPCountryWhois.csv"

INTO TABLE geo_ip

FIELDS

TERMINATED BY ","

ENCLOSED BY "\""

LINES

TERMINATED BY "\n"

(

@ip_from_string, @ip_to_string,

@ip_from, @ip_to,

@country_code, @country_string

)

SET

id := NULL,

ip_from := @ip_from,

ip_to := @ip_to,

ip_poly := GEOMFROMWKB(POLYGON(LINESTRING(

/* clockwise, 4 points and back to 0 */

POINT(@ip_from, -1), /* 0, top left */

POINT(@ip_to, -1), /* 1, top right */

POINT(@ip_to, 1), /* 2, bottom right */

POINT(@ip_from, 1), /* 3, bottom left */

POINT(@ip_from, -1) /* 0, back to start */

))),

country_code := @country_code,

country_name := @country_string

;

' | mysql --local-infile=1 -D temp

mysqldump --opt temp geo_ip > geoip.sql

echo -e "\ndone generating geoip.sql"

win7 system restore … aaarrrrgggh

July 29, 2011Ryanjunctions, system restore, user directory, win7

Warning: this is mainly me just ranting about Microsoft stupidity and a warning for anyone using junctions.

About a month or two ago I built myself a new PC (old one was dying) and installed Win7 on it. Since I was previously still on WinXP there were a few things to get used to, but all in all I enjoyed the update. There were of course some strange Windows pitfalls involved, moving your users directory to a different drive really isn’t straightforward, but works if you use junctions to link the directories.

Since Virtualbox drops the guests in a subfolder of the users directory I decided to move my users directory to one of the larger mechanical disks in my system, to keep the windows drive (a SSD disk) with plenty of space.

Now fast forward to yesterday, where some software I installed decided to clear out most of my Start->All Programs folder, leaving me with lots of installed softawre, and no Start Menu links. After trying some stuff out I finally caved in and used System Restore to get my Start Menu working again. System Restore actually worked fine, I got my Start Menu back … unfortunately it also corrupted my profile, and while googling to find out how to fix it I found a whole lot of people with the same problem.

Junction + User directory + System Restore = corrupt user, non repairable. Hello Microsoft, this is all original system software, no 3rd party stuff, breaking my user is totally uncool.

So after trying all kinds of crazy tips I found I gave up and followed the official instructions (a.k.a. You are fucked, haha bye bye user registry). And right now I am copying stuff from my old profile to my new one and setting up all the software that had stored stuff in the registry. This time my Profile stays on C: without any junctions, it’s easier to repair Virtualbox if it breaks over the images being in a non-standard path than worrying about windows corrupting my whole profile if I touch System Restore.