SHIFT

--- Sjoerd Hooft's InFormation Technology ---

User Tools

Site Tools


syslogesxi
Differences

This shows you the differences between two versions of the page.

Link to this comparison view

syslogesxi [2013/04/20 12:45] (current)
sjoerd created
Line 1: Line 1:
 += Configure Syslog on ESXi 5.1 =
  
 +Configuring syslog on ESXi 5.1 is a two step plan. First configure the ESXi host to use the syslog server and then open the firewall to allow for outgoing syslog traffic.
 +
 += Configure The Syslog Host =
 +Select the host and go to the Configuration tab, then in the Software panel click the Advanced Settings button and in the pop-up window go to Syslog. Fill in the syslog server in the "​Syslog.global.logHost"​ advanced setting:
 +<​code>​
 +udp://​sysloghost.local:​514
 +</​code>​
 +{{esxisyslog01.jpg}} \\
 +
 += Configure The Firewall =
 +To allow for outgoing syslog traffic go again to the Configuration tab and in the Software panel click the Security Profile. You'll see the current firewall settings and in the right the option to configure the Properties. In the pop-up window you can check the syslog checkbox and click OK: \\
 +{{esxisyslog02.jpg}} \\
 +
 += Limiting the Amount of Logging =
 +Now what you'll notice is that ESXi 5.1 logs enormous amounts of syslog, for my humble test hosts with about 30-40 VMs and not that much going on I could reach easily over 200000 logs a day, sometimes topping to 400000 log messages in one day. I wasn't prepared for that amount of data so I quickly decided I had to do something about that, and that led to some serious lessons:
 +# First of all, syslog implementation is not finished
 +# You can't configure it like you're used to
 +# Syslog greps into the normal logging data stream and is not a separate logging process
 +# You can only configure the log level for hostd and vpxa
 +# And even these don't work completely
 +
 +Conclusion, it is a hard lesson and completely a manual task to limit the amount of logging, so let's start with the easiest setting.
 +
 +== Limiting the SysLog Logging Level for Hostd and Vpxa ==
 +You can limit the amount of logging by going into the advanced settings again, then go to Config -> HostAgent -> Log. \\
 +Set the following settings:
 +<​code>​
 +Config.HostAgent.log.level to warning (default = verbose)
 +Vpx.Vpxa.config.log.level to warning (default = verbose)
 +</​code>​
 +
 +You've now set the syslog level to warning, but let's continue with editing the amount of logging for again these processes, the rhttpproxy and fdm.
 +
 +== Setting LogLevels on the ESXi Host ==
 +
 +> **NOTE: CHANGING THE SETTINGS AND FILES BELOW IS NOT SUPPORTED. ONLY DO SO IF DIRECTED SO BY VMWARE**
 +
 +Setup a SSH session to the host and log in.
 +
 +=== HOSTD ===
 +
 +> **NOTE: CHANGING THE SETTINGS AND FILES BELOW IS NOT SUPPORTED. ONLY DO SO IF DIRECTED SO BY VMWARE**
 +
 +Edit the file below to setup logging for the hostd process:
 +<​code>​
 +/​etc/​vmware/​hostd # vi config.xml
 +</​code>​
 +Original file log setings:
 +<code xml>
 +  <log>
 +    <​directory>/​var/​log/​vmware/</​directory>​
 +    <​level>​warning</​level>​
 +    <​maxFileNum>​8</​maxFileNum>​
 +    <​maxFileSize>​524288</​maxFileSize>​
 +    <​name>​hostd</​name>​
 +    <​outputToConsole>​false</​outputToConsole>​
 +    <​outputToFiles>​false</​outputToFiles>​
 +    <​outputToSyslog>​true</​outputToSyslog>​
 +    <​syslog>​
 +      <​facility>​local4</​facility>​
 +      <​ident>​Hostd</​ident>​
 +      <​logHeaderFile>/​var/​run/​vmware/​hostdLogHeader.txt</​logHeaderFile>​
 +    </​syslog>​
 +</​code>​
 +Add or change the lines below:
 +<code xml>
 +  <log>
 +    <​level>​warning</​level>​
 +    <​syslog>​
 +    <​level>​warning</​level>​
 +    </​syslog>​
 +</​code>​
 +
 +=== VPXA ===
 +
 +> **NOTE: CHANGING THE SETTINGS AND FILES BELOW IS NOT SUPPORTED. ONLY DO SO IF DIRECTED SO BY VMWARE**
 +
 +Edit the file below to setup logging for the hostd process:
 +<​code>​
 +/​etc/​vmware/​vpxa # vi vpxa.cfg
 +</​code>​
 +Original file log setings:
 +<code xml>
 +  <log>
 +    <​level>​warning</​level>​
 +    <​maxFileNum>​10</​maxFileNum>​
 +    <​maxFileSize>​1048576</​maxFileSize>​
 +    <​memoryLevel>​verbose</​memoryLevel>​
 +    <​outputToConsole>​false</​outputToConsole>​
 +    <​outputToFiles>​false</​outputToFiles>​
 +    <​outputToSyslog>​true</​outputToSyslog>​
 +    <​syslog>​
 +      <​facility>​local4</​facility>​
 +      <​ident>​Vpxa</​ident>​
 +      <​logHeaderFile>/​var/​run/​vmware/​vpxaLogHeader.txt</​logHeaderFile>​
 +    </​syslog>​
 +</​code>​
 +Add or change the lines below:
 +<code xml>
 +  <log>
 +    <​level>​warning</​level>​
 +    <​syslog>​
 +      <​level>​warning</​level>​
 +    </​syslog>​
 +</​code>​
 +
 +=== RHTTPPROXY ===
 +
 +> **NOTE: CHANGING THE SETTINGS AND FILES BELOW IS NOT SUPPORTED. ONLY DO SO IF DIRECTED SO BY VMWARE**
 +
 +Edit the file below to setup logging for the hostd process:
 +<​code>​
 +/​etc/​vmware/​rhttpproxy # vi config.xml
 +</​code>​
 +Original file log setings:
 +<code xml>
 +   <​log>​
 +      <!-- controls where rolling log files are stored -->
 +      <​directory>/​var/​log/​vmware/</​directory>​
 +
 +      <!-- name of log file -->
 +      <​name>​rhttpproxy</​name>​
 +
 +      <!-- controls whether logger sends its output to console also -->
 +      <​outputToConsole>​false</​outputToConsole>​
 +
 +      <!-- If true, log to files on disk -->
 +      <​outputToFiles>​false</​outputToFiles>​
 +
 +      <!-- default size(in bytes) of each log file before rolling over to next -->
 +      <​maxFileSize>​524288</​maxFileSize>​
 +
 +      <!-- default number of log files to rotate amongst -->
 +      <​maxFileNum>​8</​maxFileNum>​
 +
 +      <!-- default log level -->
 +      <​level>​verbose</​level>​
 +
 +      <!-- If true, logs to syslog -->
 +      <​outputToSyslog>​true</​outputToSyslog>​
 +
 +      <!-- syslog configuration. ​ Only used if outputToSyslog is true. -->
 +      <​syslog>​
 +         <​!-- syslog identifier to use when logging -->
 +         <​ident>​Rhttpproxy</​ident>​
 +
 +         <​!-- syslog facility to use when logging -->
 +         <​facility>​local4</​facility>​
 +
 +         <​!-- The section header contents are placed in this file at startup.
 +              When vmsyslogd rotates the hostd log file, it logs the content of this
 +              file to syslog. This is being done so that we don't lose context on log
 +              rotations.
 +              IMPORTANT: Value needs to match that in onrotate entry in
 +              vmsyslog.d/​hostd.conf
 +         ​-->​
 +         <​logHeaderFile>/​var/​run/​vmware/​rhttpproxyLogHeader.txt</​logHeaderFile>​
 +      </​syslog>​
 +   </​log>​
 +</​code>​
 +Add or change the lines below:
 +<code xml>
 +      <!-- default log level -->
 +      <​level>​warning</​level>​
 +</​code>​
 +
 +=== FDM ===
 +
 +> **NOTE: CHANGING THE SETTINGS AND FILES BELOW IS NOT SUPPORTED. ONLY DO SO IF DIRECTED SO BY VMWARE**
 +
 +Edit the file below to setup logging for the hostd process:
 +<​code>​
 +/​etc/​opt/​vmware/​fdm # vi fdm.cfg
 +</​code>​
 +Original file log setings:
 +<code xml>
 +  <log>
 +    <​level>​verbose</​level>​
 +    <​outputToFiles>​false</​outputToFiles>​
 +    <​maxFileSize>​0</​maxFileSize>​
 +    <​maxFileNum>​0</​maxFileNum>​
 +    <​memoryLevel>​trivia</​memoryLevel>​
 +    <​outputToConsole>​false</​outputToConsole>​
 +    <!-- If true, logs to syslog -->
 +    <​outputToSyslog>​true</​outputToSyslog>​
 +    <!-- syslog configuration. ​ Only used if outputToSyslog is true. -->
 +    <​syslog>​
 +      <!-- syslog identifier to use when logging -->
 +      <​ident>​Fdm</​ident>​
 +      <!-- syslog facility to use when logging -->
 +      <​facility>​local4</​facility>​
 +      <!-- The section header contents are placed in this file at startup.
 +           When vmsyslogd rotates the fdm log file, it logs the content of this
 +           file to syslog. This is being done so that we don't lose context on log
 +           ​rotations.
 +           ​IMPORTANT:​ Value needs to match that in onrotate entry in
 +           ​vmsyslog.conf.d/​fdm.conf
 +       ​-->​
 +      <​logHeaderFile>/​var/​run/​vmware/​fdmLogHeader.txt</​logHeaderFile>​
 +    </​syslog>​
 +  </​log>​
 +</​code>​
 +Add or change the lines below:
 +<code xml>
 +<​level>​warning</​level>​
 +</​code>​
 +
 +=== Add Comments===
 +If required add comments like this, do not use "#"​ to comment, this will break the services:
 +<code xml>
 +    <!--
 +    20121109 Sjoerd Changed default level from verbose to warning
 +    -->
 +</​code>​
 +
 +== Restarting all Services ==
 +
 +Don't forget to restart all services on the ESXi host and the best way to do is running this command:
 +<​code>​
 +services.sh restart
 +</​code>​
 +
 +> Note that the command will take about 5 minutes to complete and that your host will be unavailable in vCenter in the mean time. Just wait patiently for the server to come available again. This will have no impact on your VMs, unless you have [[http://​kb.vmware.com/​selfservice/​microsites/​search.do?​cmd=displayKC&​docType=kc&​docTypeID=DT_KB_1_1&​externalId=1000163|automatic startup/​shutdown]] enables which could be the case if you upgraded from previous versions. ​
 +> I've seen HA kick in, although always failed to do anything. Maybe you could stop HA before you do this. 
 +
 += Filtering Messages =
 +Even though the amount of messages should be drastically lowered, they still come with lot, in my case about 60000 a day which is still too much. I came across this message:
 +<​code>​
 +Unable to parse maxram/​maxrampercpu/​minrampercpu value:
 +</​code>​
 +According to [[http://​communities.vmware.com/​message/​2053791|this thread]] VMware support states:
 +
 +> This is a known cosmetic issue and can be safely ignored as there is no underlying issue with the license being used, it could be resolved in the next release update.
 +
 +This means you probably want to filter this in your log collector. I have ManageEngine Eventlog Analyze on which you can create the filter like this:
 +* Go to Settings
 +* Go to Database Filters
 +* Click on New Filter
 +** Give a descriptive name
 +** In the "Drop the logs containing:"​ fill in, without the quotes but including ":":​
 +*** "​unable to parse maxrampercpu value:"​
 +* Click Next
 +* Select the hosts where you want to enable the filter, which would preferably be a group including all your ESXi hosts
 +* Click Finish
 +
 +Now the amount of logging is reduced to about 5000-8000 messages a day, which is still a lot but a hell less then before. Of course you can create more filters when required, or reduce the log levels of more processes using the method described above.
 +
 += Resources =
 +[[http://​kb.vmware.com/​kb/​2003322|Configuring syslog on ESXi 5.0]] \\
 +[[http://​kb.vmware.com/​kb/​1038578|Configuring advanced options for ESX/ESXi]] \\
 +[[http://​kb.vmware.com/​kb/​1004795|Increasing Log Level]] \\
 +
 +{{tag>​syslog vmware}}
syslogesxi.txt ยท Last modified: 2013/04/20 12:45 by sjoerd