IIS Log Visualization with GoAccess

GoAccess is a slick log viewer which allows you to drill down into web server log files easily and quickly. However, different IIS versions implement different columns in their log files, which can make parsing difficult. A little pre-processing and column mapping later, we've got our IIS server logs running correctly.

GoAccess can take raw log text as piped input. I like this approach since it allows you to manipulate your log files slightly before visualizing them, but you don't have to change your log files themselves. I used awk to remove all the comment lines from the logfiles as they were consumed.

awk -F='\n' '!/^($|[:space:]*#)/{print $1}' *.log

Here, we're grabbing every line in every log file in the current directory and unless it begins with whitespace or the "#" character, we print it. We'll save this command for later – it will be used to prepare the log data for GoAccess.

Next, we'll want to configure the column properties for GoAccess. Some IIS configurations have different column mappings or orders, so this will probably be custom for you. Here's how I created my mapping.

First, I started with this excellent script from soheilpro on GitHub. Essentially, this allows you to explicitly map each column (or "field") in your IIS logs to the appropriate code consumed by GoAccess. The final configuration will look something like this:

%d %t %^ %m %U %q %^ %^ %h %u %R %s %^ %^ %L

GoAccess will use this configuration to understand how many columns there are, and what the mapping for each column should be. The mappings are documented in the GoAccess manpage.

To use this script, I first copied the "Fields" line from one of my IIS log files. It looked like this for me:

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken

This line is just a list of fields in order as they appear in the logfile. Many of these have a mapping code that GoAccess will use. For example, the field cs-method will map to the GoAccess code for "method": %m.

I modified the script from soheilpro to make sure that all my logs' Field values had a mapping. The modified script looked like this for me:

#!/usr/bin/env sh
while read line; do
  if [[ $line == \#Fields:* ]]; then
    line=${line/\#Fields: /}
    line=${line/date/%d}
    line=${line/time/%t}
    line=${line/s-ip/%^}
    line=${line/cs-method/%m}
    line=${line/cs-uri-stem/%U}
    line=${line/cs-uri-query/%q}
    line=${line/s-port/%^}
    line=${line/cs-username/%^}
    line=${line/c-ip/%h}
    line=${line/cs(User-Agent)/%u}
    line=${line/cs(Referer)/%R}
    line=${line/sc-status/%s}
    line=${line/sc-substatus/%^}
    line=${line/sc-win32-status/%^}
    line=${line/time-taken/%L}
    echo $line
    exit;
  fi
done

After the script was modified, I ran it against one of my log files to get the resulting GoAccess configuration:

cat u_ex190201.log | ../go-access-configuration.sh

This gave me the configuration we saw earlier:

%d %t %^ %m %U %q %^ %^ %h %u %R %s %^ %^ %L

Now we have all the pieces, and we can pipe them together. We'll first use the awk command to remove all the comments from our log files, then pipe the raw log text into the GoAccess command which will use the configuration string we just generated. Viola:

awk -F='\n' '!/^($|[:space:]*#)/{print $1}' *.log | goaccess - -o report.html --log-format "%d %t %^ %m %U %q %^ %^ %h %u %R %s %^ %^ %L" --date-format "%Y-%m-%d" --time-format "%H:%M:%S"

You'll then have a nice report ready in a report.html file in your current directory.

Kudos to the GoAccess team for a great tool!

~ <3 Ike