Skip to main content
Submitted by Edward on Fri, 07/29/2011 - 18:10

So I was reading a post on the Emerging Threats signature mailing list and someone mentioned the SANS Internet Storm Center's 404 Project where your webserver sends information to SANS ISC whenever it receives a request for something that it can't answer, a 404 error.

The recommended code on the site is to embed some PHP code into your custom 404 inside apache that will POST data to the ISC website. I got to thinking that for hosting providers or people who want to control what the send, as well as not have the webserver process this request directly, why not pull this data from the Apache access log.

Below is my attempt to do exactly that. I don't currently have an ISC user id or ISC user key so I can't actually test this. And my Apache logs with 404 aren't very interesting anyway

Plus, I don't know if Johannes Ullrich is even interested in this type of data going back or not in real-time.

Here it is anyway

use strict;
use warnings;
use LWP;
use Getopt::Long;
use Pod::Usage;
use Data::Dumper;
use MIME::Base64;

# Create a new useragent to submit information with.
my $ua = LWP::UserAgent->new(); 

# Define variables so we can use them later.
my (@files, $key, $user, $man, $help);
my $OptResult = GetOptions(
                            "help"   =>    \$help,
                            "man"    => \$man,
                            "user=s" => \$user,
                            "key=s"  => \$key,
                            "file=s@" => \@files,

# Display the help information if the user requests it, or doesn't supply
# their ISC userid and userkey along with a list of files
pod2usage( -verbose => 1 ) if ($help);
pod2usage( -verbose => 2 ) if ($man);
if (!defined($user) && !defined($key) && !defined($files[0])) {
    pod2usage( -verbose => 1 );
=head1 NAME

404Project - Used to send Apache 404 logs to SANS Internet Storm Center 


Usage: $0 [--help] [--man] --file apche_access.log --user ISC_ID --key ISC_User_Key
=head1 OPTIONS

=over 8

=item B<-h>

Print a brief help message and exit.

=item B<-f LogFile>

Point to the log file you wish to send 404 from.

=item B<-u ISCUserID>

This is your Internet Storm Center ID

=item B<-k ISCUserKey>

This is the key associated with your ISC_ID.



B<> will read the given input log file(s) parse them and send the
log entries with a 404 status to the SANS Internet Storm Center 404 Project


my %months = (
  'Jan' => '01',
  'Feb' => '02',
  'Mar' => '03',
  'Apr' => '04',
  'May' => '05',
  'Jun' => '06',
  'Jul' => '07',
  'Aug' => '08',
  'Sep' => '09',
  'Oct' => '10',
  'Nov' => '11',
  'Dec' => '12'

foreach my $file (@files) {
    open(FILE, "<$file") || die "Unable to read file: $file: $!\n";
    while(<FILE>) {
        if ($_ =~ m!^(\d+\.\d+\.\d+\.\d+) - .+ \[(..)/(...)/(....):(..:..:..) .....\] "(GET|POST) (.*) HTTP/..." 404 \d+ "[^"]+" "([^"]+)"!) {
            my $reqMon   = $months{$3};
            my $sData = $user . $key . $7 . $1 . $8 . $4 . "-" . $reqMon . "-" . $2 . $5;
            if ($sData !~ m/^$/) {
                $sData = encode_base64($sData); 
            print "Submitting: \"$4-$reqMon-$2 $5\" $1 $7\n";
            my $result = $ua->post("$user&version=1",    Content => $sData);
            if($result->is_success) {
                print "URL OK\n";
            } elsif ($result->is_redirect) {
                print "REDIRECT\n";
                print Dumper($result);
            } else {
                print "Other...\n";
                print Dumper($result);


DISCLAIMER: This is a personal Web site, produced in my own time and solely reflecting my personal opinions. Statements on this site do not represent the views or policies of my employer, past or present, or any other organization with which I may be affiliated. All content is copyrighted.