Programming – Page 4 – Ryan Schulze

How to add debugging to shellscripts

May 4, 2011September 27, 2014Ryan1 Commentbash, debug

Debugging bash scripts is pretty straightforward, throwing around a couple echo and set -x quickly gives you what you need. But what if you want to add a nice breakpoint, debugging to lots of paces in the code or turn all debugging on or off at once? Then this little script I wrote is the right thing for you: debug.sh just download it to the same directory as your script and include it with the following line:

. debug.sh

1	. debug.sh

It contains 4 simple functions that will make your bash coding easier.
debug and breakpoint both print the argument with a timestamp to STDERR
You can turn off all the functions by adding a DEBUG=false into your code

Example:

#!/bin/bash
. debug.sh

debug_on
echo foo
debug_off

debug "I am a test"

breakpoint "wait for a bit"

echo done

#!/bin/bash

. debug.sh

debug_on

echo foo

debug_off

debug "I am a test"

breakpoint "wait for a bit"

echo done

Output:

+ echo foo
foo
+ debug_off
+ true
+ set +x
[16:37:47] I am a test
[16:37:47] -breakpoint- wait for a bit
press any key to continue

done

+ echo foo

foo

+ debug_off

+ true

+ set +x

[16:37:47] I am a test

[16:37:47] -breakpoint- wait for a bit

press any key to continue

done

Disemvoweling

May 2, 2011May 13, 2013Ryancss, disemvoweling, javascript

Talk about weird words … ok, according to Wikipedia disemvoweling is the term for replacing or removing vowels from words. Commonly used as a tool for moderating. I’m pretty sure everyone has run across certain disemvoweled words on the internet like f*ck or sh*t. Anyway I went and made a pure html/javascript page that does just that, removes any vowels from an inputted text. The usefullness can certainly be argued, it was more for me to brush up on my javascript and css skills.

http://www.dopefish.de/projects/disemvoweling/

captcha cracking

March 18, 2011May 13, 2013Ryan1 Commentcaptcha, gocr, linux, php

This is a pretty old posting from 2009 I just recently discovered in my “drafts” directory. Nowadays there are probably easier and more elegant ways of defeating a captcha, but for old times sake, here is my simple approach.
———————–

Eclectic and Marko were so kind as to “provide” me a captcha to play around with. Took me a few days of poking around and googling but in the end it was easier than I had thought. As long as there aren’t and logic errors in the code (e.g. bad or no session handling) you probably won’t get around some kind of OCR. As OCR software I decided to use gocr because it is free, runs under linux, and it is fairly easy to train to specific needs. Because I knew which libraries were being used to create the captcha images, it was possible for me to build a testing area. This just speeds things up a bit, the process would have worked just as well off the original website. First off: the spambot in action -> http://captcha.dopefish.de/spambot.php, and the website it accesses: http://captcha.dopefish.de/

Now I’ll describe the steps I took to defeat the captcha. Look at what happens on failed and successful inputs, first write a script that works if you enter the solution manually. I used the following 2 php functions for getting and posting stuff (and keeping the session intact)

$ch = curl_init();

function get_url( $ch, $url, $timeout = 5 )
{
    $url = str_replace( "&amp;", "&amp;", urldecode(trim($url)) );

    $cookie = tempnam ("/tmp", "CURLCOOKIE");
    curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );
    curl_setopt( $ch, CURLOPT_URL, $url );
    curl_setopt( $ch, CURLOPT_COOKIEFILE, $cookie );
    curl_setopt( $ch, CURLOPT_COOKIEJAR, $cookie );
    curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
    curl_setopt( $ch, CURLOPT_ENCODING, "" );
    curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
    curl_setopt( $ch, CURLOPT_AUTOREFERER, true );
    curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false );    # required for https urls
    curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
    curl_setopt( $ch, CURLOPT_TIMEOUT, $timeout );
    curl_setopt( $ch, CURLOPT_MAXREDIRS, 10 );
    $content = curl_exec( $ch );
    $response = curl_getinfo( $ch );

    return array( $content, $response );
}

function post_url( $ch, $url,  $postvars, $timeout = 5 )
{
    $url = str_replace( "&amp;", "&amp;", urldecode(trim($url)) );

    $cookie = tempnam ("/tmp", "CURLCOOKIE");
    curl_setopt( $ch, CURLOPT_POST ,1);
    curl_setopt( $ch, CURLOPT_POSTFIELDS , $postvars);
    curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );
    curl_setopt( $ch, CURLOPT_URL, $url );
    curl_setopt( $ch, CURLOPT_COOKIEFILE, $cookie );
    curl_setopt( $ch, CURLOPT_COOKIEJAR, $cookie );
    curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );
    curl_setopt( $ch, CURLOPT_ENCODING, "" );
    curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
    curl_setopt( $ch, CURLOPT_AUTOREFERER, true );
    curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false );    # required for https urls
    curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );
    curl_setopt( $ch, CURLOPT_TIMEOUT, $timeout );
    curl_setopt( $ch, CURLOPT_MAXREDIRS, 10 );
    $content = curl_exec( $ch );
    $response = curl_getinfo( $ch );

    return array( $content, $response );
}

$ch = curl_init();

function get_url( $ch, $url, $timeout = 5 )

{

$url = str_replace( "&", "&", urldecode(trim($url)) );

$cookie = tempnam ("/tmp", "CURLCOOKIE");

curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );

curl_setopt( $ch, CURLOPT_URL, $url );

curl_setopt( $ch, CURLOPT_COOKIEFILE, $cookie );

curl_setopt( $ch, CURLOPT_COOKIEJAR, $cookie );

curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );

curl_setopt( $ch, CURLOPT_ENCODING, "" );

curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );

curl_setopt( $ch, CURLOPT_AUTOREFERER, true );

curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false ); # required for https urls

curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );

curl_setopt( $ch, CURLOPT_TIMEOUT, $timeout );

curl_setopt( $ch, CURLOPT_MAXREDIRS, 10 );

$content = curl_exec( $ch );

$response = curl_getinfo( $ch );

return array( $content, $response );

}

function post_url( $ch, $url, $postvars, $timeout = 5 )

{

$url = str_replace( "&", "&", urldecode(trim($url)) );

$cookie = tempnam ("/tmp", "CURLCOOKIE");

curl_setopt( $ch, CURLOPT_POST ,1);

curl_setopt( $ch, CURLOPT_POSTFIELDS , $postvars);

curl_setopt( $ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; rv:1.7.3) Gecko/20041001 Firefox/0.10.1" );

curl_setopt( $ch, CURLOPT_URL, $url );

curl_setopt( $ch, CURLOPT_COOKIEFILE, $cookie );

curl_setopt( $ch, CURLOPT_COOKIEJAR, $cookie );

curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, true );

curl_setopt( $ch, CURLOPT_ENCODING, "" );

curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );

curl_setopt( $ch, CURLOPT_AUTOREFERER, true );

curl_setopt( $ch, CURLOPT_SSL_VERIFYPEER, false ); # required for https urls

curl_setopt( $ch, CURLOPT_CONNECTTIMEOUT, $timeout );

curl_setopt( $ch, CURLOPT_TIMEOUT, $timeout );

curl_setopt( $ch, CURLOPT_MAXREDIRS, 10 );

$content = curl_exec( $ch );

$response = curl_getinfo( $ch );

return array( $content, $response );

}

Now train a gocr database for the images. Obviously it get’s better the more you train it.
Since curl is taking care of session handling, we can use the get_url() function for downloading the captcha image. I pipe it through this shell command to make it easier for gocr to read:

convert \
-quality 100 \
-blur 0.7 \
-level 9%,60%,100.0 \
-monochrome \
-negate \
-blur 0.1 \
-gamma 1 \
-depth 8 \
-compress none \

convert \

-quality 100 \

-blur 0.7 \

-level 9%,60%,100.0 \

-monochrome \

-negate \

-blur 0.1 \

-gamma 1 \

-depth 8 \

-compress none \

It turnes this:

into this:

Since the valid captcha result is always the same length, we can check if gocr matched all the chars. If it looks good we can use post_url() to continue our session and throw all the fields at the form and submit it. See, wasn’t that hard. Most of the time is spent training gocr and converting the image into something easier to read. It doesn’t solve 100% of the images, more like 80-90%, but still better than nothing ;-).

Wireshark remote capturing

March 8, 2011May 13, 2013Ryanbash, remote, tcpdump, wireshark

yeah, this is real simple stuff, not really worth writing a script for it. but on the other hand it saves me from remembering how to do it every time I need it (which isn’t often). So here is a little script to setup remote capturing with wireshark.
All it basically does is ssh to the remote host and tcpdump sucking the output via stdout through the ssh connection to a local pipe, that is then used by wireshark to display the stream. Because of this you may want to make sure you aren’t capturing your own ssh data when doing this 😉

#!/bin/bash

destination="1.2.3.4" # server to run the remote capture on
filter="port 80"      # tcpdump filter options
interface="eth0"      # interface to listen on on remote server
opts="-n -p -s 0"     # other tcmpdump options

##### let the fun begin #####

mypipe="/tmp/remotecap.$$.cap"
mkfifo ${mypipe}

ssh root@${destination} "tcpdump ${opts} -i ${interface} -w - ${filter}" &gt; ${mypipe} &amp;
pipepid=$!

wireshark -k -N ntC -t a -i ${mypipe}

kill ${pipepid}
rm -f ${mypipe}

#!/bin/bash

destination="1.2.3.4" # server to run the remote capture on

filter="port 80" # tcpdump filter options

interface="eth0" # interface to listen on on remote server

opts="-n -p -s 0" # other tcmpdump options

##### let the fun begin #####

mypipe="/tmp/remotecap.$$.cap"

mkfifo ${mypipe}

ssh root@${destination} "tcpdump ${opts} -i ${interface} -w - ${filter}" > ${mypipe} &

pipepid=$!

wireshark -k -N ntC -t a -i ${mypipe}

kill ${pipepid}

rm -f ${mypipe}

MySQL selecting IPs via CIDR

February 23, 2011December 15, 2017Ryancidr, inet_aton, inet_ntoa, ip, myysql, network

Quick little snippet here for selecting IPs from a database based off a CIDR subnet. First off a table structure with some test data:

mysql> desc tmp;
+-------+------------------+
| Field |     Type         |
+-------+------------------+
| ip1   | int(12) unsigned |  
| ip2   | varchar(15)      |
+-------+------------------+

mysql> select * from tmp;
+------------+-----------------+
| ip1        |             ip2 |
+------------+-----------------+
| 16909060   | 1.2.3.4         |
| 167904004  | 10.2.3.4        |
| 2915086080 | 173.192.175.0   |
| 2915086100 | 173.192.175.20  |
| 2915086335 | 173.192.175.255 |
+------------+-----------------+

mysql> desc tmp;

+-------+------------------+

| Field | Type |

+-------+------------------+

| ip1 | int(12) unsigned |

| ip2 | varchar(15) |

+-------+------------------+

mysql> select * from tmp;

+------------+-----------------+

| ip1 | ip2 |

+------------+-----------------+

| 16909060 | 1.2.3.4 |

| 167904004 | 10.2.3.4 |

| 2915086080 | 173.192.175.0 |

| 2915086100 | 173.192.175.20 |

| 2915086335 | 173.192.175.255 |

+------------+-----------------+

Now let’s say we want all IPs from the subnet 173.192.175.16/28, using a simple 173.192.175.% would provide false results since you don’t want the whole /24.

If your IP is stored as an unsigned int (good for you) than you can use this snippet to search for matching IPs:

select inet_ntoa(ip1) as ip1,ip2 from tmp where (ip1 & power(2, 32) - power(2, (32 - 28))) = inet_aton('173.192.175.16');

1	select inet_ntoa(ip1) as ip1,ip2 from tmp where (ip1 & power(2, 32) - power(2, (32 - 28))) = inet_aton('173.192.175.16');

If your IP is stored as a varchar (for whatever reason), the only difference is a inet_aton() around the IP field.

select inet_ntoa(ip1) as ip1,ip2 from tmp where (inet_aton(ip2) & power(2, 32) - power(2, (32 - 28))) = inet_aton('173.192.175.16');

1	select inet_ntoa(ip1) as ip1,ip2 from tmp where (inet_aton(ip2) & power(2, 32) - power(2, (32 - 28))) = inet_aton('173.192.175.16');

No matter which one you use, the result will be:

+----------------+----------------+
| ip1            |         ip2    |
+----------------+----------------+
| 173.192.175.20 | 173.192.175.20 |
+----------------+----------------+

+----------------+----------------+

| ip1 | ip2 |

+----------------+----------------+

| 173.192.175.20 | 173.192.175.20 |

+----------------+----------------+