CyberHosting

Updating LogFormat for better stats collection

As mentioned here collecting stats for Cyberpanel accesslog is a real pita.
https://forums.cyberpanel.net/discussion/1021/reporting-on-unique-visitors-to-a-website-wordpress#latest

I submitted fix to cyberpanel for this in 1.9.4 but this only fixes it for new domains created after the server is running it.
https://github.com/usmannasir/cyberpanel/pull/235

Please note the server should be upgraded to this version before doing any of the 2 or optional 3rd step.

This will upgrade to 1.9.4
sh <(curl -s https://raw.githubusercontent.com/usmannasir/cyberpanel/v1.9.4/preUpgrade.sh || wget -qO - https://raw.githubusercontent.com/usmannasir/cyberpanel/v1.9.4/preUpgrade.sh)

The information below is for how to update existing vhost and ols/LS LogFormat to the same universal standard cPanel uses which bring a lot of cool possibilities to getting stats.

Steps

1. Updating httpd.conf for new log format for stats.

grep LogFormat /usr/local/lsws/conf/httpd.conf

Fix httpd.conf logformat inline
sed -i "s|^LogFormat.*|LogFormat '%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"' combined|g" /usr/local/lsws/conf/httpd.conf

Before:
[[email protected]:~]# grep LogFormat /usr/local/lsws/conf/httpd.conf
LogFormat '"%v %h %l %u %t \"%r\" %>s %b"' combined
[[email protected]:~]#

After:
[[email protected]:~]# grep LogFormat /usr/local/lsws/conf/httpd.conf
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
[[email protected]:~]#

Example:
[[email protected]:~]# sed -i "s|^LogFormat.*|LogFormat '%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"' combined|g" /usr/local/lsws/conf/httpd.conf
[[email protected]:~]#


2. Fixing vhost log format for a single vhost.

What this does is insert the line "LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined" above the lines with "CustomLog" in vhost conf file

Before:
CustomLog /home/example.com/logs/example.com.access_log combined


After:
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined

Fixing a single vhost full example:

Commands:
Test without editing
sed "s/.*CustomLog.*/ LogFormat '%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"' combined\n&/g" /usr/local/lsws/conf/vhosts/example.com/vhost.conf

Edit inline
sed -i "s/.*CustomLog.*/ LogFormat '%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"' combined\n&/g" /usr/local/lsws/conf/vhosts/example.com/vhost.conf


Before:
[[email protected]:~]# cat /usr/local/lsws/conf/vhosts/example.com/vhost.conf
<VirtualHost *:80>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
CustomLog /home/example.com/logs/example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>

</VirtualHost>

<VirtualHost *:443>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
CustomLog /home/example.com/logs/example.com.access_log combined
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
AddHandler application/x-httpd-php73 .php
</VirtualHost>
[[email protected]:~]#


After:
[[email protected]:~]# cat /usr/local/lsws/conf/vhosts/example.com/vhost.conf
<VirtualHost *:80>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>

</VirtualHost>

<VirtualHost *:443>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
AddHandler application/x-httpd-php73 .php
</VirtualHost>
[[email protected]:~]#


Testing Example:
[[email protected]:~]# sed "s/.*CustomLog.*/ LogFormat '%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"' combined\n&/" /usr/local/lsws/conf/vhosts/example.com/vhost.conf
<VirtualHost *:80>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>

</VirtualHost>

<VirtualHost *:443>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
AddHandler application/x-httpd-php73 .php
</VirtualHost>
[[email protected]:~]#


3. Bulk editing all hosts to have new log format. This is great if you have alot of domains on cyberpanel server and do not want to have to do this one time per site.

Test on all vhosts without editing
sudo find /usr/local/lsws/conf/vhosts/ -type f -name 'vhost.conf' -exec sed "s/.*CustomLog.*/ LogFormat '%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"' combined\n&/g" {} \;

Apply to all vhosts via inline editing
sudo find /usr/local/lsws/conf/vhosts/ -type f -name 'vhost.conf' -exec sed -i "s/.*CustomLog.*/ LogFormat '%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"' combined\n&/g" {} \;


Testing Example:
[[email protected]:~]# sudo find /usr/local/lsws/conf/vhosts/ -type f -name 'vhost.conf' -exec sed "s/.*CustomLog.*/ LogFormat '%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"' combined\n&/g" {} \;
<VirtualHost *:80>

ServerName dev.example.com
ServerAlias www.dev.example.com
ServerAdmin [email protected]
SuexecUserGroup devuser devuser
DocumentRoot /home/dev.example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/dev.example.com/logs/dev.example.com.access_log combined
AddHandler application/x-httpd-php73 .php .php7 .phtml
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>

</VirtualHost>

<VirtualHost *:443>

ServerName dev.example.com
ServerAlias www.dev.example.com
ServerAdmin [email protected]
SuexecUserGroup devuser devuser
DocumentRoot /home/dev.example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/dev.example.com/logs/dev.example.com.access_log combined
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/dev.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/dev.example.com/privkey.pem
AddHandler application/x-httpd-php73 .php .php7 .phtml
</VirtualHost>
<VirtualHost *:80>

ServerName wcloud.example.com
ServerAlias www.wcloud.example.com
ServerAdmin [email protected]
SuexecUserGroup wcloudw wcloudw
DocumentRoot /home/wcloud.example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/wcloud.example.com/logs/wcloud.example.com.access_log combined
AddHandler application/x-httpd-php73 .php .php7 .phtml
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>

</VirtualHost>

<VirtualHost *:443>

ServerName wcloud.example.com
ServerAlias www.wcloud.example.com
ServerAdmin [email protected]
SuexecUserGroup wcloudw wcloudw
DocumentRoot /home/wcloud.example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/wcloud.example.com/logs/wcloud.example.com.access_log combined
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/wcloud.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/wcloud.example.com/privkey.pem
AddHandler application/x-httpd-php73 .php .php7 .phtml
</VirtualHost>
<VirtualHost *:80>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>

</VirtualHost>

<VirtualHost *:443>

ServerName example.com
ServerAlias www.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/example.com.access_log combined
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/example.com/privkey.pem
AddHandler application/x-httpd-php73 .php
</VirtualHost>
<VirtualHost *:80>

ServerName my.example.com
ServerAlias www.my.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html/my.example.com
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/my.example.com.access_log combined
AddHandler application/x-httpd-php73 .php
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>

</VirtualHost>
<VirtualHost *:443>

ServerName my.example.com
ServerAlias www.my.example.com
ServerAdmin [email protected]
SuexecUserGroup user user
DocumentRoot /home/example.com/public_html/my.example.com
LogFormat '%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-Agent}i"' combined
CustomLog /home/example.com/logs/my.example.com.access_log combined
<IfModule LiteSpeed>
CacheRoot lscache
</IfModule>
SSLEngine on
SSLVerifyClient none
SSLCertificateFile /etc/letsencrypt/live/my.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/my.example.com/privkey.pem
AddHandler application/x-httpd-php73 .php
</VirtualHost>
[[email protected]:~]#


Applying changes to all vhost.confs by editing inline.
[[email protected]:~]# sudo find /usr/local/lsws/conf/vhosts/ -type f -name 'vhost.conf' -exec sed -i "s/.*CustomLog.*/ LogFormat '%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"' combined\n&/g" {} \;
[[email protected]:~]#

If all goes well you will want to restart the lsws webservice to activate changes
[[email protected]:~]# service lsws restart
Redirecting to /bin/systemctl restart lsws.service
[[email protected]:~]# service lsws status
Redirecting to /bin/systemctl status lsws.service
● lshttpd.service - LiteSpeed HTTP Server
Loaded: loaded (/usr/lib/systemd/system/lshttpd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2020-02-03 09:42:22 EST; 4s ago
Process: 27822 ExecStop=/usr/local/lsws/bin/lswsctrl delay-stop (code=exited, status=0/SUCCESS)
Process: 27834 ExecStart=/usr/local/lsws/bin/lswsctrl start (code=exited, status=0/SUCCESS)
Main PID: 27844 (litespeed)


I hope you enjoyed this post and it helps get your access_logs in a usable format.

My next post will show how you can get cool stats once you have log format changed into the new format.
Sign In or Register to comment.
CyberPanel Discord

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!