Live Log Searching

From Roaring Penguin
Revision as of 12:50, 27 June 2017 by MCoyne (talk | contribs)

Jump to: navigation, search

CanIt provides powerful log searching tools available to the administrators through the WebUI using Administration->Search Logs. This feature requires that the installation already have the Log Collelator package installed. This is provided on Hosted CanIt, but on-site implementations require that it be installed following the instructions in chapter 6.1 of the Installation Guide.

All SMTP logs are indexed so that they can be searched using a wide variety of fields and a wide variety of matching conditions. However, this indexing process can take up to 30 minutes to complete. If you require immediate access to logs, they are available from the commandline. If you are on Hosted CanIt, only Roaring Penguin staff have access to the live logs, so you should address your query to us. For those with their own installation, here is some basic log searching tips:

Log Location

If the Archiver package has not yet been installed, mail will be in Debian's default log location:


Once installed, the Archiver breaks the logs up into date-based files in the format:


There is also a symbolic link which is updated during nightly cron tasks which points to the current day's logs:


Viewing Logs in Real-Time

The easiest way to see current mail flow is to use the 'tail' command with the -f option:

   $ tail -f /var/log/mail-daily/current.log

This will display all log lines as they are added to the file. In order to stop viewing the logs you must give the termination command by hitting Ctrl+C, or send the task to the background with Ctrl+Z (it can be pulled to the foreground with %#, where # is the task number which was output after Ctrl+Z).

You can restrict which lines get displayed by piping the output through the 'grep' command as below. This would look like:

   $ tail -f /var/log/mail-daily/current.log | grep search-term

Searching the Logs

The commandline logs are searchable using the 'grep' command. The standard format for this command is:

   $ grep search-term file-to-search

for example:

   $ grep error /var/log/mail-daily/current.log

which would return only lines that have 'error' in them up to the time that the command was executed.

This command is case sensitive unless you add the -i option. Another useful option is -v which provides the inverse of the search (ie. everything that does NOT contain the search term). The search term should be a single string. If you need to search for a space you should enclose the search term in quotes. Here is an example with all of these options:

   $ grep -iv "long search term" /var/log/mail-daily/current.log

which will return anything that does not contain the string "long search term", regardless of capitalization. You can learn more about the grep command with:

   $ man grep

Use 'q' to exit the manual.

Once you have found an entry and would like to find all logs related to that transaction you can copy the QueueID and grep using it. The QueueID is the unique string, starting with a lowercase letter such as: 'v1GMFT4c009743:'. Thus, to find all logs for that transaction you would use:

   $ grep v1GMFT4c009743 /var/log/mail-daily/current.log

The output of 'grep' can further be pipe'd to other commands, including further 'grep' commands to restrict the search even more. For example:

   $ grep /var/log/mail-daily/current.log | grep "Subject Here"

However, grep only searches on a line-by-line basis and will only match if a given line contains both of those terms. Different indicators like Senders, Recipients, Subjects etc. are often not contained on the same line.