Monitoring your network with nmap

December 15, 2008 at 10:56 pm

This all came about while I was pondering the best way to dump an overview of what was sat on my network in the datacenter; the main reason being a lack of useful documentation telling me what IPs were in use to narrow it down at all. The idea was that it’d dump in CSV format something along the lines of:

IP,TCP Ports
10.0.0.1,22 143 5000 12345 31337

Of course, once I had this data, my mind started down the path of ‘wouldn’t it be nice to have an automated system that would e-mail me and tell me if any ports change (read: “if any developers set something up without my knowledge”), and also tell me if any hosts are added to our network.  I present to you an automated network monitoring script, this will scan your network and send you a detailed report (with port probing), via a direct SMTP connection to your remote mail server - which is useful in the event that your network gets compromised and you can no longer trust any binaries.  All that you need to do is put it somewhere, make it executable, and add it to a cronjob - mine runs hourly, but I’m paranoid.

0 */1 * * * /root/bin/nmap.pl auto >/dev/null 2>&1

The e-mails look something similar to this:

Nmap Scan Report. Hosts/Ports changed!
Generated on: 2008-12-08 18:39:44

——–BEGIN——–
10.0.0.1 is a new host with 3 ports open/filtered.

10.0.0.1 has these ports open: 22,111,5000
22 might be ssh
111 might be rpcbind
5000 might be vtun
——–END——–

Creative Commons LicenseAll code that I’ve written is licensed under the Creative Commons Attribution-Non-Commercial-Share Alike 2.0 UK: England & Wales Licence.  Where non-commercial means don’t sell it, not ‘you can’t use it in your company’. =]

?Download nmap.pl
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
#!/usr/bin/perl
use File::Copy;
use Nmap::Parser;
use Net::DNS::Sendmail;
use warnings;
use strict;
 
#config begin
my $nmap_path = "/usr/bin/nmap"; #path to nmap binary
my $nmap_args = "-T5 -F -sV"; #nmap arguments (accepts multiple), do not specify any output arguments!
my $ips = ""; #networks to scan, accepts standard nmap input
my $save_as = "/root/parser-cache.xml"; #path to storage file - this file is used for comparative scans, so it's best not to place it in /tmp.
my $email_from_address = ''; #from address to use when sending mail - should probably exist to avoid sender callouts
my $email_to_address = ''; #address you want results delivered to
my $email_sender_domain = ''; #domain to masquerade as - useful in cases where remote mta checks helo.
my $email_subject = "Nmap Network Scan Results"; #subject for email.
my $debug = "false"; #enables some output if you need to test auto on the cli - i.e. why mail isn't being sent.
#config end
 
my $np = new Nmap::Parser;
my $old = new Nmap::Parser;
my $curr = new Nmap::Parser;
my $tempstore = "/tmp/parser-cache.tmp";
my $emailfile = "/tmp/nmap-notify";
my $newmachines = 0;
my $auto = (lc($ARGV[0]) eq "auto");
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst)=localtime(time);
 
sub debug
{
    print "@_\n" if($debug eq "true");
}
 
sub print_twice
{
    print "@_";
    print EMAIL "@_";
}
 
sub create_files
{
    foreach (@_)
        {
            if(!-e $_)
                {
                    print "$_ does not exist... creating\n" if(!$auto);
                    open(TEMP,">$_") or die "Can't create $_: $!";
                    close(TEMP) or die "Couldn't close $_: $!";
                }
        }
}
 
create_files($save_as,$tempstore);
 
if($auto)
{
    open(EMAIL,">$emailfile") or die "Can't create $emailfile: $!";
    print EMAIL "Nmap Scan Report.  Hosts/Ports changed!\nGenerated on: ";
    printf EMAIL "%4d-%02d-%02d %02d:%02d:%02d\n\n",$year+1900,$mon+1,$mday,$hour,$min,$sec;
    print EMAIL "--------BEGIN--------\n";
}
 
if($auto)
{
    debug("Running auto mode");
 
    if(-s $save_as)
    {
        $old->parsefile($save_as); #load previous state
        $curr->cache_scan($tempstore); #set temporary file to store xml after we're done
        $curr->parsescan($nmap_path, $nmap_args, $ips); #scan current hosts
 
        for my $ip ($curr->get_ips('up'))
        {
            my $ip_old = $old->get_host($ip);
            my $ip_curr = $curr->get_host($ip);
            my @alltcp = $ip_curr->tcp_open_ports();
            my $numtcp = $#alltcp + 1;
            if(!$ip_old)
            {
                print_twice("$ip is a new host with".$numtcp."ports open/filtered.\n");
                ++$newmachines;
            }
        }
 
        print_twice("\n") if($newmachines != 0);
 
        for my $ip ($curr->get_ips('up'))
        {
            my $ip_old = $old->get_host($ip);
            my $ip_curr = $curr->get_host($ip);
            my %port = ();
 
            if($ip_old)
            {
                my @tcpdiff = grep { $port{$_} < 2}
                                    (map {$port{$_}++; $_}
                                    ( $ip_curr->tcp_open_ports , $ip_old->tcp_open_ports));
 
                if(scalar @tcpdiff)
                {
                    print_twice("$ip has these new ports open: ".join(',',@tcpdiff)."\n");
                    for (@tcpdiff)
                    {
                        print_twice("\t"."$_ might be ",$ip_curr->tcp_service($_)->name,"\n");
                    }
                    print_twice("\n");
                }
            }else{
                my @tcp = (map {$port{$_}++; $_}
                            ( $ip_curr->tcp_ports('open') ));
 
                if(scalar @tcp)
                {
                    print_twice("$ip has these ports open: ".join(',',@tcp)."\n");
                    for (@tcp)
                    {
                        print_twice("\t"."$_ might be ",$ip_curr->tcp_service($_)->name,"\n");
                    }
                    print_twice("\n");
                }
            }
        }
    }else{
        $curr->cache_scan($tempstore); #set temporary file to store xml after we're done
        $curr->parsescan($nmap_path, $nmap_args, $ips); #scan current hosts
 
        for my $ip ($curr->get_ips('up'))
        {
            my $ip_curr = $curr->get_host($ip);
            my @alltcp = $ip_curr->tcp_open_ports();
            my $numtcp = $#alltcp + 1;
            print_twice("$ip is a new host with ".$numtcp." ports open/filtered.\n");
            ++$newmachines;
        }
 
        print_twice("\n") if($newmachines != 0);
 
        for my $ip ($curr->get_ips('up'))
        {
            my $ip_curr = $curr->get_host($ip);
            my %port = ();
 
            my @tcp = (map {$port{$_}++; $_}
                      ( $ip_curr->tcp_ports('open') ));
 
            if(scalar @tcp)
            {
                print_twice("$ip has these ports open: ".join(',',@tcp)."\n");
 
                for (@tcp)
                {
                    print_twice("\t"."$_ might be ",$ip_curr->tcp_service($_)->name,"\n");
                }
 
                print_twice("\n");
            }
        }
    }
    move($tempstore, $save_as);
    print EMAIL "--------END--------";
    close EMAIL;
 
    open(EMAIL,"<$emailfile") or die "Can't read $emailfile: $!";
    my $mail = Net::DNS::Sendmail->new();
    $mail->verbose() if($debug eq "true");
    $mail->senderdomain( "$email_sender_domain" );
    $mail->to( "$email_to_address" );
    $mail->from( "$email_from_address" );
    $mail->subject( "$email_subject" );
    while () { $mail->data($_); }
    $mail->sendmail() if($newmachines != 0);
    debug("mail sent");
    close EMAIL;
} else {
    #Unsure whether or not to move the output of this into $save_as... it may
    #be that someone runs this, and doesn't take care of the output, in which
    #case, we still want to be notified that night.
    $np->cache_scan($save_as);
    $np->parsescan($nmap_path, $nmap_args, $ips);
    print "IP,OPEN TCP PORTS\n";
    for my $ip ($np->get_ips('up'))
    {
        my $ip_check = $np->get_host($ip);
        my %port = ();
 
        my @tcp = (map {$port{$_}++; $_}
                  ( $ip_check->tcp_ports('open') ));
 
        print "$ip,".join(' ',@tcp)."\n" if(scalar @tcp);
    }
}

Embedding subtitles into avi files on the linux command line

May 24, 2008 at 10:59 pm

After a few hours of googling, and turning up pretty much nothing; I figured I ought to make a note of this somewhere. I often find myself in the situation where I’ve just downloaded some asian movie (I’m a huge fan of Asian Extreme -think along the lines of Old Boy, Ishii the killer, etc-, and, I really want to be able to watch what I’ve downloaded on my TV) and it doesn’t have the subtitles embedded in the avi file (and no, before you suggest it, I will NOT watch dubbed crap). Microsoft, unfortunately, doesn’t think anyone in the world would want to watch something with subtitles, so this isn’t a feature of the Xbox360. My Xbox is hooked up to a NAS box which has all my music/tv/movies on it, and this box runs a /very/ minimal install of ubuntu - ~150mb in total.

So, down to business. You’ll need transcode installed, along with mplayer, then it’s a simple case of:

transcode -i videofile.avi -x mplayer=-sub subfile.xxx -o outputfile.avi -y xvid

There are other output formats, but I use xvid, since it’s supported by the Xbox. It takes a while longer than just straight transcoding, but it’s worth it to not have to watch it on my laptop.

Handling the transfer of 25tb of data

May 24, 2008 at 10:51 pm

So, your RAID becomes slightly unreliable, you’ve spent 2 months using rsync to transfer 25m files off the failing array, and having rebuilt it, you want to transfer those files back, in a faster way. How to go about it?

The above is a problem that I’m currently dealing with, and I found something of a hacky, yet elegant solution to this. I’m a huge believer in lftp, it’s a brilliant piece of software, and happily lends itself to many different situations. lftp also supports permissions/ownership setting on files it creates, which is a key feature in handling the above problem; along with all of that, it parallelizes transfers particularly well, meaning that small files aren’t stalled because of large many-gb files slowly transferring.

Unfortunately, I discovered that lftp doesn’t handle directory ownership/permissions _at all_. My original idea was just to set off lftp, walk away and in a few days marvel at the 25tb that had been transferred; however, I really needed to maintain the perms across the board. rsync came into play again:

rsync -avz --stats --progress --include "*/" --exclude "*" ip.address.goes.here:/path/to/files/* /path/

The above command creates just the directories that exist on the other end, skipping all files - this sets up the structure that I need for lftp to work.

Once that’s completed, it’s a simple case of:

lftp sftp://root@ip.address.here -e "mirror -c --parallel=50 --allow-chown --allow-suid /path/to/files ./"