Question


Can I use log management tools for monitoring the VCD system in OnApp?

Environment


OnApp with VMware Cloud Director integration

Answer


OnApp has some very detailed, verbose logging set by default, which allows you to easily troubleshoot errors or just monitor the status of your VCD system within OnApp. A side-effect of this is that the log files can get very large – and therefore, it can be a good idea to use a logging tool to assist and parse the logs in a structured way.

The instruction below describes how to integrate the OnApp logs into a free, hosted log management tool called Papertrail.

Get Papertrail

The first step would be to sign up for an account at Papertrail and log in to your dashboard. Here you need to add your first system. The system you will be adding at this point will be your OnApp controller, so log into that via SSH and run the commands that it lists. Alternatively, there is a one-line command you can run at the top of this page that will automate much of the initial setup. Below is an example of how it should look.

wget -qO - --header="X-Papertrail-Token: xhxhASnX5nucRPFrH9XZ" \
  https://papertrailapp.com/destinations/3809456/setup.sh | sudo bash
CODE


Once this has run, it will automatically add your OnApp CP to the systems. However, you need to configure remote_syslog to ensure you can gather all of the OnApp logs that are required.

Install and Configure remote_syslog

1. Download the latest remote_syslog release at https://github.com/papertrail/remote_syslog2/releases 

2. Unpack the download:

tar xzf ./remote_syslog*.tar.gz
CODE


3. Open the remote_syslog directory:

cd remote_syslog
CODE


4. Copy the remote_syslog application to the /usr/local/bin directory: 

sudo cp ./remote_syslog /usr/local/bin
CODE


5. Download the following config file and edit the Papertrail host and port to be the same as you use in your dashboard:

https://github.com/jimfreeman/onapp_vcd_papertrail/blob/master/log_files.yml

This file should be located at /etc/log_files.yml

6. Download the following file and place it into /etc/init.d/remote_syslog:

https://github.com/papertrail/remote_syslog2/blob/master/examples/remote_syslog.init.d

7. Run the following command to give it the correct permissions:

chmod +x /etc/init.d/remote_syslog
CODE


8. Run the following command to make it launch on startup:

update-rc.d remote_syslog defaults
CODE


9. Start the service:

service remove_syslog start
CODE


From now, if you go to the dashboard of the system in Papertrail, you should see your live logs being processed.


Saving Searches and Alerts

Now all of the logs are being processed by Papertrail, then the next step is to configure searches and alert email address(es) based on the search results. Several examples are provided below.

Daily Resync: Each night at 1 am, OnApp runs a resync of the OnApp and VCD environment to ensure all assets inside of OnApp are up to date and the same as in VMware Cloud Director. To retrieve the search criterion based on this, use the following search query:

program:production_vcloud.log AND "Daily resync"
CODE


Performance metric tracking: By default, OnApp runs an L2 stats transaction every 3 minutes (this is also customizable). If this transaction does not run correctly, there will be blank entries in performance metrics for virtual servers within your environment. The following search criterion will output all L2 stat collections to make sure they have run successfully. We recommend setting the frequency of this alert to one day.

program:production_vcloud_stats_runner.log AND "[L2]"
CODE


Failures during import or resync procedure: This search criterion will look for any failures during the import, such as a user not importing due to not having an email address assigned, or a user being from an external directory source. You can then configure this to be sent to you via an alert and resolve any failures that have happened.

program:production_vcloud.log AND “failure”
CODE