Today, I decided that I needed to see a little bit more about the traffic hitting my website. So I set off attempting to find a tool that will help me do just that. After doing some Googling, I ran across a Microsoft utility called Log Parser. So I downloaded and installed it. A great post to read about Log Parser is by Steve Bunting. Another post by Jeff Atwood here. Microsoft Technet has an article about how Log Parser works here. Log Parser is not officially a supported Microsoft tool, the forum site can be found here. Another article from Symantec discusses the fields for parsing and some examples here. I played around with Log Parser some and got some rudimentary results. But since I didn’t want to recreate the wheel, I decided to keep searching.

After Googling on “IIS7 Statistics”, I stumbled upon AWStats. I decided to install it. The documentation for installing on Windows Server 2008 is pretty weak. I found a  great guide to installing AWStats here. The steps can be summarized as follows:

1. Install ActivePerl here.
2. Install AWStats here. Be sure to change the default installation path to: “C:\AWStats”. If you let it go into ‘Program Files’, you’ll run into trouble in step 5 below as the path with ‘Program Files’ has a space. Trust me, I tried it. I’m sure there’s a work-around, (please post your comments about a solution!) but after spending 30 minutes with various quotes, etc. I gave just up and moved ahead!
3. Create a new default configuration file.
4. Edit the default configuration file with notepad. Run notepad as Administrator.
5. Change entry: LogFile=”/var/log/httpd/mylog.log” to: LogFile=”c:\awstats\tools\logresolvemerge.pl c:\inetpub\logs\LogFiles\W3SVC3\*.log |” **Make sure double quotes are not formatted type, ***Be sure to get the right site number at the end of ‘W3SVC#’
6. Change LogFormat = 2 (I had already changed the logformat in IIS7 to W3C when I set up the site, so no additional changes were needed to the LogFormat)
7. Verify SiteDomain and HostAliases vars.
8. Open Command Prompt (cmd) with Administrative Privileges and navigate to “c:\AWStats\wwwroot\cgi-bin” and run “awstats.pl -config=www.yourdomain.com” (don’t add the prefix “awstats” or suffix “conf” it will add it for you)

So, at this point, I’m getting an error about the format of the log file. It doesn’t match the format it’s expecting. My second attempt at generating the stats database was to use the LogFormat string as follows from my guiding post above:

LogFormat=”%time2 %other %method %url %other %query %other %host %other %code % %other %other %bytesd”

This also produced the same type of error. I then opened up IIS7 Manager and navigated to the Webserver Node, selected logging. I then chose the ‘Select Fields’ option. I noticed here, that there are many more fields selected than what is defined in the LogFormat string I’m providing. Here is the list for fields currently being logged and a screen shot showing them as well:

Date, Time, Client IP Address, User Name, Service Name, Server Name, Server IP Address, Server Port, Method, URI Stem, URI Query, Protocol Status, Protocol Substatus, Win32 Status, Bytes Sent, Bytes Received, Time Taken, Protocol Version, Host, User Agent, Cookie, Referer

My next task is to see if I can manually determine the actual format of the log file. So I Googled on “W3SVC IIS7 file format” and got a nice explanation of file formats here. I navigated to “C:\inetput\logs\LogFiles\W3SVC3” and opened up the current log file. I immediately noticed that there are #Fields: statements in the file. Here is a sample line from the log file:

#Fields: date time s-sitename s-computername s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs-version cs(User-Agent) cs(Cookie) cs(Referer) cs-host sc-status sc-substatus sc-win32-status sc-bytes cs-bytes time-taken

I then compared the commented #Fields with actual content in the log file, and they appear to match. I then cross-referenced the following list of fields from the awstats config file:

 

#   %host             Client hostname or IP address (or Sender host for mail log)
#   %host_r           Receiver hostname or IP address (for mail log)
#   %lognamequot      Authenticated login/user with format: “john”
#   %logname          Authenticated login/user with format: john here
#   %time1            Date and time with format: [dd/mon/yyyy:hh:mm:ss +0000] or [dd/mon/yyyy:hh:mm:ss]
#   %time2            Date and time with format: yyyy-mm-dd hh:mm:ss
#   %time3            Date and time with format: Mon dd hh:mm:ss or Mon dd hh:mm:ss yyyy
#   %time4            Date and time with unix timestamp format: dddddddddd
#   %methodurl        Method and URL with format: “GET /index.html HTTP/x.x”
#   %methodurlnoprot  Method and URL with format: “GET /index.html”
#   %method           Method with format: GET
#   %url              URL only with format: /index.html
#   %query            Query string (used by URLWithQuery option)
#   %code             Return code status (with format for web log: 999)
#   %bytesd           Size of document in bytes
#   %refererquot      Referer page with format: “http://from.com/from.htm”
#   %referer          Referer page with format: http://from.com/from.htm
#   %uabracket        User agent with format: [Mozilla/4.0 (compatible, …)]
#   %uaquot           User agent with format: “Mozilla/4.0 (compatible, …)”
#   %ua               User agent with format: Mozilla/4.0_(compatible…)
#   %gzipin           mod_gzip compression input bytes: In:XXX
#   %gzipout          mod_gzip compression output bytes & ratio: Out:YYY:ZZpct.
#   %gzipratio        mod_gzip compression ratio: ZZpct.
#   %deflateratio     mod_deflate compression ratio with format: (ZZ)
#   %email            EMail sender (for mail log)
#   %email_r          EMail receiver (for mail log)
#   %virtualname      Web sever virtual hostname. Use this tag when same log
#                     contains data of several virtual web servers. AWStats
#                     will discard records not in SiteDomain nor HostAliases
#   %cluster          If log file is provided from several computers (merged by
#                     logresolvemerge.pl), use this to define cluster id field.
#   %extraX           Another field that you plan to use for building a
#                     personalized report with ExtraSection feature (See later).
#   If your log format has some fields not included in this list, use:
#   %other            Means another not used field
#   %otherquot        Means another not used double quoted field

After mapping, here is my new LogFormat file:

 

LogFormat=”%time2 %other %other %host_r %method %url %query %other %logname %host %other %ua %other %referer %other %code %other %other %bytesd %other %other”

A couple notes here about why step 8, the statistics build, didn’t work. First, in step 5, I was using “W3SVC1” instead of “W3SVC3”. The number at the end indicates the site. The site I’m using is site 3. Second, I selected additional fields to log several days ago. This changed the format of the file from its original state. So, the build program had problems recognizing the older record formats. To solve this, I moved all of the old log files out into another directory so that only the files with the newest format are being used in the database.

I now continued on, repeating from step 8 above:

1. Open Command Prompt (cmd) with Administrative Privileges and navigate to “c:\AWStats\wwwroot\cgi-bin” and run “awstats.pl -config=www.yourdomain.com” **If it runs successfully, you will get Phase 1 …, Phase 2 …, Jumped lines …, Parsed lines …, …, Found 804 new qualified records.
2. Run “awstats.pl -config=www.yourdomain.com –output –staticlinks > ..\yourdomain_com_stats.html” **remember to use “..\” to build the html file in directory above.

At this point, I used Windows explorer and navigated to “C:\AWStats\wwwroot\” and clicked on the generated html file. I expected something magical to appear, but was disappointed to see it display:

WARNING: LastLine parameter in history file is ‘20100426184511’ so in future. May be you need to correct manually the line LastLine in some awstats*.www.larrybrouwer.com.conf files. Create/Update database for config “C:\AWStats\wwwroot\cgi-bin/awstats.www.larrybrouwer.com.conf” by AWStats version 6.95 (build 1.943) From data in log file “c:\awstats\tools\logresolvemerge.pl c:\inetpub\logs\LogFiles\W3SVC3\*.log |”… Phase 1 : First bypass old records, searching new record… Searching new records from beginning of log file… Jumped lines in file: 0 Parsed lines in file: 804 Found 0 dropped records, Found 0 corrupted records, Found 804 old records, Found 0 new qualified records.

I then Googled “AWStats Warning LastLine parameter in history file” and found numerous posts, one of which is here. It appears that the IIS7 outputs timestamps in GMT, and my server is on US Central Time (GMT-0600), so I need to correct for that as indicated here. There is a parameter in the AWStats config file:

LoadPlugin=”timezone –6”

I tried –6 first, but got the same result. A Wikipedia explanation here is useful. Daylight savings is the culprit, so I changed the parameter to “–7”, and it worked! I’ve temporarily put it at https://www.larrybrouwer.com:81/stats/larrybrouwer_com_stats.html for viewing. Sadly, clicking on this link won’t work since DNN is re-writing the URL. Bummer! It’ll be changing the link in the near future anyway when I integrate it into my DotNetNuke site.

Additional Steps to work on:

1. Set it up to view from my DNN website
2. Have it automatically produce new stats
3. Have it automatically compensate for Daylight Savings

(see my post, Adding AWStats to DotNetNuke)

One final note… While trying to integrate AWStats into DNN, I stumbled upon the logging features in DNN. Logging is not turned on by default now as is outlined in this post here. Per this post, to enable logging, as administrator, navigate to: Site Settings \ Advanced Settings \ Host Settings \ Site Log History and change it to a positive number representing the number of days you wish to have logs kept.

To view the logs, click on Admin then select Site Log in the Basic Features section.

Update on 4/27/2010:

I’ve noticed that the gifs aren’t showing up in the viewed pages. So I went poking around through the installation document that comes with the software. It’s located at c:\AWStats\docs\awstats_setup.html. It probably would have been best to follow these instructions first as it describes in detail my current problems. It also discusses how to set up the log file just for AWStats use (i.e. disabling all the unnecessary fields).

So I’m going to fix my installation for IIS7 using the documentation that came with the software.

1. Configure IIS7 to create logs in the “Extended W3C log format”. I’m leaving my log file formats the way they are. Since I’ve already re-mapped the fields, no need to change them again.
2. Copy the contents of AWStats cgi-bin folder to your server’s cgi-bin directory. In my case I have two websites running. So I created new cgi-bin dirs for each site.
3. Copy the icon directory and sub-directory contents to the target website root folders. Again, I copied the icon folder to my two sites.
4. Create a configuration file. I’m using the one I’ve already configured.
5. Edit your configuration file. LogFile values remain the same; LogType=”W”, stays the same; LogFormat stays the same; Change the DirIcons parameter to reflect the relative path of icon directory. In my case, I don’t have to change the location since I’ve now moved the icon dir to the proper location off the root directory.

Now the icons are showing up fine.

Looking more closely at the AWStats report, I’ve noticed that there links to other reports that aren’t there. Doing some more looking at the documentation, I see that there are many more reports that can be generated. A quick way to build up all the reports is to use awstats_buildstaticpages.pl. Here’s the script I created: