SHIFT

--- Sjoerd Hooft's InFormation Technology ---

User Tools

Site Tools


Sidebar

Sponsor:

Would you like to sponsor this site?
Or buy me a beer?:


Recently Changed Pages:

View All Pages
View All Q Pages


View All Tags


Sign up for Q to post comments.





WIKI Disclaimer: As with most other things on the Internet, the content on this wiki is not supported. It was contributed by me and is published “as is”. It has worked for me, and might work for you.
Also note that any view or statement expressed anywhere on this site are strictly mine and not the opinions or views of my employer.


Terms And Conditions for Q users


Pages with comments

PageDateDiscussionTags
2019/06/22 10:36 1 Comment
2019/03/15 16:02 1 Comment
2019/03/15 16:02 1 Comment
2019/03/15 16:02 3 Comments
2017/04/20 15:28 1 Comment
2017/04/20 15:23 1 Comment
2017/04/19 14:44 1 Comment
2017/04/17 20:10 1 Comment
2017/04/17 20:07 1 Comment
2017/04/17 19:58 1 Comment
2017/04/17 19:52 1 Comment

View All Comments

awstats

AWStats - Statistics for your Webserver

In this page we extend our Apache Reverse Proxy with a statistics tool called awstats, which is nothing more than a collection of perl scripts and tools.

Download and Default Installation

I did this with version 7.0 which I downloaded as a gzip/tar package for linux systems. If you run the script from a non-standard location you'll get this warning:

Warning: AWStats standard directory on Linux OS is '/usr/local/awstats'.
If you want to use standard directory, you should first move all content
of AWStats distribution from current directory:
/tmp/awstats/awstats-7.0
to standard directory:
/usr/local/awstats
And then, run configure.pl from this location.
Do you want to continue setup from this NON standard directory [yN] ?

In that case, answer with “N” and move the files to the standard location '/usr/local/awstats'. Then run the script with this command:

reverseproxy:/usr/local/awstats/tools # perl awstats_configure.pl

A script will be started which will configure some of the basic configuration:

----- AWStats awstats_configure 1.0 (build 1.9) (c) Laurent Destailleur -----
This tool will help you to configure AWStats to analyze statistics for
one web server. You can try to use it to let it do all that is possible
in AWStats setup, however following the step by step manual setup
documentation (docs/index.html) is often a better idea. Above all if:
- You are not an administrator user,
- You want to analyze downloaded log files without web server,
- You want to analyze mail or ftp log files instead of web log files,
- You need to analyze load balanced servers log files,
- You want to 'understand' all possible ways to use AWStats...
Read the AWStats documentation (docs/index.html).

-----> Running OS detected: Linux, BSD or Unix

-----> Check for web server install

Enter full config file path of your Web server.
Example: /etc/httpd/httpd.conf
Example: /usr/local/apache2/conf/httpd.conf
Example: c:\Program files\apache group\apache\conf\httpd.conf
Config file path ('none' to skip web server setup):
> /etc/apache2/httpd.conf

-----> Check and complete web server config file '/etc/apache2/httpd.conf'
  Add 'Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"'
  Add 'Alias /awstatscss "/usr/local/awstats/wwwroot/css/"'
  Add 'Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"'
  Add 'ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"'
  Add '<Directory>' directive
  AWStats directives added to Apache config file.

-----> Update model config file '/usr/local/awstats/wwwroot/cgi-bin/awstats.model.conf'
  File awstats.model.conf updated.

-----> Need to create a new config file ?
Do you want me to build a new AWStats config/profile
file (required if first install) [y/N] ? y

-----> Define config file name to create
What is the name of your web site or profile analysis ?
Example: www.mysite.com
Example: demo
Your web site, virtual server or profile name:
> www.warmetal.nl

-----> Define config file path
In which directory do you plan to store your config file(s) ?
Default: /etc/awstats
Directory path to store config file(s) (Enter for default):
>

-----> Create config file '/etc/awstats/awstats.www.warmetal.nl.conf'
 Config file /etc/awstats/awstats.www.warmetal.nl.conf created.

-----> Restart Web server with '/sbin/service httpd restart'
service: no such service httpd

-----> Add update process inside a scheduler
Sorry, configure.pl does not support automatic add to cron yet.
You can do it manually by adding the following command to your cron:
/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.warmetal.nl
Or if you have several config files and prefer having only one command:
/usr/local/awstats/tools/awstats_updateall.pl now
press ENTER to continue...


A SIMPLE config file has been created: /etc/awstats/awstats.www.warmetal.nl.conf
You should have a look inside to check and change manually main parameters.
You can then manually update your statistics for 'www.warmetal.nl' with command:
> perl awstats.pl -update -config=www.warmetal.nl
You can also read your statistics for 'www.warmetal.nl' with URL:
> http://localhost/awstats/awstats.pl?config=www.warmetal.nl

press ENTER to finish...

reverseproxy:/usr/local/awstats/tools #

Post Installation Configuration

Httpd.conf to virtualhost

First we move and edit the adjustments in the httpd.conf to a separate config file in the vhosts directory. This was added to the httpd.conf file:

#
# Directives to allow use of AWStats as a CGI
#
Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"
Alias /awstatscss "/usr/local/awstats/wwwroot/css/"
Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"
ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"

#
# This is to permit URL access to scripts/files in AWStats directory.
#
<Directory "/usr/local/awstats/wwwroot">
    Options None
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

I removed this and added an awstats.conf file in the /etc/apache2/vhosts.d directory:

<VirtualHost *:80>
    ServerAdmin sjoerd_ @ _ warmetal.nl
    ServerName webstats.warmetal.nl

        DocumentRoot /usr/local/awstats/wwwroot
        ErrorLog /var/log/apache2/warmetalwebstats-error_log
        CustomLog /var/log/apache2/warmetalwebstats-access_log combined

        HostnameLookups Off
        UseCanonicalName Off
        ServerSignature On

        DirectoryIndex awstats.www.warmetal.nl.html
        #
        # Directives to allow use of AWStats as a CGI
        #
        Alias /awstatsclasses "/usr/local/awstats/wwwroot/classes/"
        Alias /awstatscss "/usr/local/awstats/wwwroot/css/"
        Alias /awstatsicons "/usr/local/awstats/wwwroot/icon/"
        ScriptAlias /awstats/ "/usr/local/awstats/wwwroot/cgi-bin/"

        #
        # This is to permit URL access to scripts/files in AWStats directory.
        #
        <Directory "/usr/local/awstats/wwwroot">
            Options None
            AllowOverride None
            Order allow,deny
            Allow from all
        </Directory>

</VirtualHost>

AWstats Settings

After adding the vhost we needed to configure awstats itself. Below you'll find all settings from the awstats confog file. Changed values have a “# Changed xx to xx ” line behind them:

schurefsrv01:/etc/awstats # cat awstats.www.warmetal.nl.conf | grep -v '^$' | grep -v '^#'
LogFile="/var/log/apache2/access_log" # Changed to the correct access_log
LogType=W
LogFormat=1
LogSeparator=" "
SiteDomain="www.warmetal.nl"
HostAliases="www.warmetal.nl www.www.warmetal.nl 127.0.0.1 localhost"
DNSLookup=2
DirData="/var/lib/awstats"
DirCgi="/awstats"
DirIcons="/awstatsicons"
AllowToUpdateStatsFromBrowser=1 # Changed from 0 to 1
AllowFullYearView=3 # Changed from 2 to 3
EnableLockForUpdate=1 # Changed from 0 to 1
DNSStaticCacheFile="dnscache.txt"
DNSLastUpdateCacheFile="dnscachelastupdate.txt"
SkipDNSLookupFor=""
AllowAccessFromWebToAuthenticatedUsersOnly=0
AllowAccessFromWebToFollowingAuthenticatedUsers=""
AllowAccessFromWebToFollowingIPAddresses=""
CreateDirDataIfNotExists=0
BuildHistoryFormat=text
BuildReportFormat=xhtml #Changed from html to xhtml
SaveDatabaseFilesWithPermissionsForEveryone=0
PurgeLogFile=0
ArchiveLogRecords=0
KeepBackupOfHistoricFiles=0
DefaultFile="index.php index.html"
SkipHosts=""
SkipUserAgents=""
SkipFiles=""
SkipreferrersBlackList=""
OnlyHosts=""
OnlyUserAgents=""
OnlyUsers=""
OnlyFiles=""
NotPageList="css js class gif jpg jpeg png bmp ico rss xml swf"
ValidHTTPpres="200 304"
ValidSMTPpres="1 250"
AuthenticatedUsersNotCaseSensitive=0
URLNotCaseSensitive=0
URLWithAnchor=0
URLQuerySeparators="?;"
URLWithQuery=0
URLWithQueryWithOnlyFollowingParameters=""
URLWithQueryWithoutFollowingParameters=""
URLReferrerWithQuery=0
WarningMessages=1
ErrorMessages=""
DebugMessages=0
NbOfLinesForCorruptedLog=50
WrapperScript=""
DepreUA=0
MiscTrackerUrl="/js/awstats_misc_tracker.js"
LevelForBrowsersDetection=2         # 0 disables Browsers detection.
                                    # 2 reduces AWStats speed by 2%
                                    # allphones reduces AWStats speed by 5%
LevelForOSDetection=2               # 0 disables OS detection.
                                    # 2 reduces AWStats speed by 3%
LevelForRefererAnalyze=2            # 0 disables Origin detection.
                                    # 2 reduces AWStats speed by 14%
LevelForRobotsDetection=2           # 0 disables Robots detection.
                                    # 2 reduces AWStats speed by 2.5%
LevelForSearchEnginesDetection=2    # 0 disables Search engines detection.
                                    # 2 reduces AWStats speed by 9%
LevelForKeywordsDetection=2         # 0 disables Keyphrases/Keywords detection.
                                    # 2 reduces AWStats speed by 1%
LevelForFileTypesDetection=2        # 0 disables File types detection.
                                    # 2 reduces AWStats speed by 1%
LevelForWormsDetection=0            # 0 disables Worms detection.
                                    # 2 reduces AWStats speed by 15%
UseFramesWhenCGI=1
DetailedReportsOnNewWindows=1
Expires=0
MaxRowsInHTMLOutput=1000
Lang="auto"
DirLang="./lang"
ShowMenu=1
ShowSummary=UVPHB
ShowMonthStats=UVPHB
ShowDaysOfMonthStats=VPHB
ShowDaysOfWeekStats=PHB
ShowHoursStats=PHB
ShowDomainsStats=UVPHB
ShowHostsStats=PHBL
ShowAuthenticatedUsers=0
ShowRobotsStats=HBL
ShowWormsStats=0
ShowEMailSenders=0
ShowEMailReceivers=0
ShowSessionsStats=1
ShowPagesStats=PBEX
ShowFileTypesStats=HB
ShowFileSizesStats=0
ShowOSStats=1
ShowBrowsersStats=1
ShowScreenSizeStats=0
ShowOriginStats=PH
ShowKeyphrasesStats=1
ShowKeywordsStats=1
ShowMiscStats=a
ShowHTTPErrorsStats=1
ShowSMTPErrorsStats=0
ShowClusterStats=0
AddDataArrayMonthStats=1
AddDataArrayShowDaysOfMonthStats=1
AddDataArrayShowDaysOfWeekStats=1
AddDataArrayShowHoursStats=1
IncludeInternalLinksInOriginSection=0
MaxNbOfDomain = 10
MinHitDomain  = 1
MaxNbOfHostsShown = 10
MinHitHost    = 1
MaxNbOfLoginShown = 10
MinHitLogin   = 1
MaxNbOfRobotShown = 10
MinHitRobot   = 1
MaxNbOfPageShown = 10
MinHitFile    = 1
MaxNbOfOsShown = 10
MinHitOs      = 1
MaxNbOfBrowsersShown = 10
MinHitBrowser = 1
MaxNbOfScreenSizesShown = 5
MinHitScreenSize = 1
MaxNbOfWindowSizesShown = 5
MinHitWindowSize = 1
MaxNbOfRefererShown = 10
MinHitRefer   = 1
MaxNbOfKeyphrasesShown = 10
MinHitKeyphrase = 1
MaxNbOfKeywordsShown = 10
MinHitKeyword = 1
MaxNbOfEMailsShown = 20
MinHitEMail   = 1
FirstDayOfWeek=1
ShowFlagLinks=""
ShowLinksOnUrl=1
UseHTTPSLinkForUrl=""
MaxLengthOfShownURL=64
HTMLHeadSection=""
HTMLEndSection=""
MetaRobot=0
Logo="awstats_logo6.png"
LogoLink="http://awstats.sourceforge.net"
BarWidth   = 260
BarHeight  = 90
StyleSheet=""
color_Background="FFFFFF"               # Background color for main page (Default = "FFFFFF")
color_TableBGTitle="CCCCDD"             # Background color for table title (Default = "CCCCDD")
color_TableTitle="000000"               # Table title font color (Default = "000000")
color_TableBG="CCCCDD"                  # Background color for table (Default = "CCCCDD")
color_TableRowTitle="FFFFFF"    # Table row title font color (Default = "FFFFFF")
color_TableBGRowTitle="ECECEC"  # Background color for row title (Default = "ECECEC")
color_TableBorder="ECECEC"              # Table border color (Default = "ECECEC")
color_text="000000"                             # Color of text (Default = "000000")
color_textpercent="606060"              # Color of text for percent values (Default = "606060")
color_titletext="000000"                # Color of text title within colored Title Rows (Default = "000000")
color_weekend="EAEAEA"                  # Color for week-end days (Default = "EAEAEA")
color_link="0011BB"                             # Color of HTML links (Default = "0011BB")
color_hover="605040"                    # Color of HTML on-mouseover links (Default = "605040")
color_u="FFAA66"                                # Background color for number of unique visitors (Default = "FFAA66")
color_v="F4F090"                                # Background color for number of visites (Default = "F4F090")
color_p="4477DD"                                # Background color for number of pages (Default = "4477DD")
color_h="66DDEE"                                # Background color for number of hits (Default = "66DDEE")
color_k="2EA495"                                # Background color for number of bytes (Default = "2EA495")
color_s="8888DD"                                # Background color for number of search (Default = "8888DD")
color_e="CEC2E8"                                # Background color for number of entry pages (Default = "CEC2E8")
color_x="C1B2E2"                                # Background color for number of exit pages (Default = "C1B2E2")
LoadPlugin="geoip GEOIP_STANDARD /usr/local/awstats/GeoIP/GeoIP.dat"
LoadPlugin="geoip_city_maxmind GEOIP_STANDARD /usr/local/awstats/GeoIP/GeoLiteCity.dat"
ExtraTrackedRowsLimit=500

Plugins

As you can see from the awstats config file I added two plugins to the configuration. These plugins can be downloaded from:

Note: Be sure to download the binary version.

Installing them is easy, simply place them in the directory as defined in the awstats config file. Also note that these databases are updated every month so you should replace them every month with a fresh download.

For these plugins to work you'll need an additional Perl module called “Geo::IP”. That can be downloaded from here. I took the latest version which was 1.25 at the time.

Installing the plugin is easy and explained on the download website, this was my setup:

sjoerd@reverseproxy:/tmp/awstats> gunzip Geo-IP-PurePerl-1.25.tar.gz
sjoerd@reverseproxy:/tmp/awstats> tar -xf Geo-IP-PurePerl-1.25.tar
sjoerd@reverseproxy:/tmp/awstats> cd Geo-IP-PurePerl-1.25/
sjoerd@reverseproxy:/tmp/awstats/Geo-IP-PurePerl-1.25> perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for Geo::IP::PurePerl
sjoerd@reverseproxy:/tmp/awstats/Geo-IP-PurePerl-1.25> make
cp lib/Geo/IP/PurePerl.pm blib/lib/Geo/IP/PurePerl.pm
cp geoip-lookup blib/script/geoip-lookup
/usr/bin/perl "-MExtUtils::MY" -e "MY->fixin(shift)" blib/script/geoip-lookup
Manifying blib/man1/geoip-lookup.1
Manifying blib/man3/Geo::IP::PurePerl.3pm
sjoerd@reverseproxy:/tmp/awstats/Geo-IP-PurePerl-1.25> make test
PERL_DL_NONLAZY=1 /usr/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t
t/0_base..........ok
t/1_lookup........skipped
        all skipped: No GeoIP.dat found
t/2_namelookup....skipped
        all skipped: No GeoIP.dat found
All tests successful, 2 tests skipped.
Files=3, Tests=1,  0 wallclock secs ( 0.02 cusr +  0.02 csys =  0.04 CPU)
sjoerd@reverseproxy:/tmp/awstats/Geo-IP-PurePerl-1.25> sudo make install
Installing /usr/lib/perl5/site_perl/5.10.0/Geo/IP/PurePerl.pm
Installing /usr/share/man/man1/geoip-lookup.1
Installing /usr/share/man/man3/Geo::IP::PurePerl.3pm
Installing /usr/bin/geoip-lookup
Writing /usr/lib/perl5/site_perl/5.10.0/x86_64-linux-thread-multi/auto/Geo/IP/PurePerl/.packlist
Appending installation info to /usr/lib/perl5/5.10.0/x86_64-linux-thread-multi/perllocal.pod

Make sure you do the “make install” as root.

Crontab

You could consider adding the update command to the crontab as told in the installer script. I did that like this:

reverseproxy:/usr/local/awstats/tools # crontab -l
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (/tmp/crontab.XXXXcfCIRq installed on Wed Jun 15 13:55:09 2011)
# (Cron version V5.0 -- $Id: crontab.c,v 1.12 2004/01/23 18:56:42 vixie Exp $)
0 * * * * /usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=www.warmetal.nl

Restart / Reload Apache

To apply the changes reload apache: <pre bash> reverseproxy:/etc/apache2/vhosts.d # /etc/init.d/apache2 reload Reload httpd2 (graceful restart) done

or - restart:

reverseproxy:/usr/local/awstats/tools # /etc/init.d/apache2 restart [Wed Jun 15 13:49:50 2011] [warn] worker https://10.10.10.15/app already used by another worker Syntax OK Starting httpd2 (prefork) [Wed Jun 15 13:49:50 2011] [warn] worker https://10.10.10.15/app already used by another worker

                                                                                                           done

</code>

Check

You should now have a working setup on:

http://reverseproxy/awstats/awstats.pl?config=www.warmetal.nl

Troubleshooting

However, I'm used that you get a lot / a few errors when trying to use awstats:

DirData Does Not Exist

Error: AWStats database directory defined in config file by 'DirData' parameter (/var/lib/awstats) does not exist or is not writable.
Setup ('/etc/awstats/awstats.www.warmetal.nl.conf' file, web server or permissions) may be wrong.
Check config file, permissions and AWStats documentation (in 'docs' directory). 

This simply means that /var/lib/awstats has not been created yet:

sjoerd@reverseproxy:/var/lib> sudo mkdir awstats
sjoerd@reverseproxy:/var/lib> sudo chmod a+w awstats/

Couldn't Open Server Log File - Permission Denied

Error: Couldn't open server log file "/var/log/apache2/access_log" : Permission denied

This error popped up when trying the “update now” button in the interface. It means that the user that is running the awstats webserver hasn't got the permission to read this file, and that is caused because the containing directory can't be read. Fix that like this:

reverseproxy:/var/log # chmod o+rx apache2/

Other Permission Problems

Something else that might cause problems is the permissions on the awstats directory. Change them like this:

reverseproxy:/usr/local/awstats # chown -R root:root *
reverseproxy:/usr/local/awstats # chown -R wwwrun:root wwwroot/
reverseproxy:/usr/local/awstats # ll
total 24
drwxr-xr-x 2 root   root 4096 Jun 16 11:20 GeoIP
-rwxr-xr-x 1 root   root 6803 Dec  5  2010 README.TXT
drwxr-xr-x 4 root   root 4096 Dec  6  2010 docs
drwxr-xr-x 4 root   root 4096 Dec  6  2010 tools
drwxr-xr-x 7 wwwrun root 4096 Dec  6  2010 wwwroot

AWStats Resources

You could leave a comment if you were logged in.
awstats.txt · Last modified: 2013/03/26 13:17 by sjoerd