Live Log Searching

From Roaring Penguin
Jump to: navigation, search

CanIt provides powerful log searching tools available to the administrators through the WebUI using Administration->Search Logs. This feature requires that the installation already have the Log Collelator package installed. This is provided on Hosted CanIt, but on-site implementations require that it be installed following the instructions in chapter 6.1 of the Installation Guide.

All SMTP logs are indexed so that they can be searched using a wide variety of fields and a wide variety of matching conditions. However, this indexing process can take up to 30 minutes to complete. If you require immediate access to logs, they are available from the commandline. If you are on Hosted CanIt, only Roaring Penguin staff have access to the live logs, so you should address your query to us. For those with their own installation, here is some basic log searching tips:

Log Location

If the Archiver package has not yet been installed, mail will be in Debian's default log location:

   /var/log/mail.log

Once installed, the Archiver breaks the logs up into date-based files in the format:

   /var/log/mail-daily/mail-YYYY-MM-DD.log

There is also a symbolic link which is updated during nightly cron tasks which points to the current day's logs:

   /var/log/mail-daily/current.log

Viewing Logs in Real-Time

The easiest way to see current mail flow is to use the 'tail' command with the -f option:

   tail -f /var/log/mail-daily/current.log

This will display all log lines as they are added to the file. In order to stop viewing the logs you must give the termination command by hitting Ctrl+C, or send the task to the background with Ctrl+Z (it can be pulled to the foreground with %#, where # is the task number which was output after Ctrl+Z).

You can restrict which lines get displayed by piping the output through the 'grep' command as below. This would look like:

   tail -f /var/log/mail-daily/current.log | grep search-term

Searching the Logs

The commandline logs are searchable using the 'grep' command. The standard format for this command is:

   grep search-term file-to-search

for example:

   grep error /var/log/mail-daily/current.log

which would return only lines that have 'error' in them up to the time that the command was executed.

This command is case sensitive unless you add the -i option. Another useful option is -v which provides the inverse of the search (ie. everything that does NOT contain the search term). The search term also needs to be a single string, so if you want to look for something with multiple works, you will need to enclose it in quotes. Here is an example with all of these options:

   grep -iv "long search term" /var/log/mail-daily/current.log

which will return everything except lines that contain the string "long search term", regardless of capitalization. You can learn more about grep (or almost any other command) from the manual (man) pages with:

   man grep

Use 'q' to exit the manual.

Search results will only include the exact lines with that search term. For example, if you find all lines that mention a certain email address, you will not also be given the remainder of the lines related to that transaction. If you would like all logs related to that transaction you can copy the QueueID and grep using it. The QueueID is the unique string included on every line for that given transaction which starts with a lowercase letter then a mixture of alpha-numeric characters, such as: 'v1GMFT4c009743'. Thus, to find all logs for that transaction you would use:

   $ grep v1GMFT4c009743 /var/log/mail-daily/current.log

The output of 'grep' can further filtered to other commands using the pipe (|) character. This includes further 'grep' commands to restrict the search even more. For example:

   $ grep sender@example.com /var/log/mail-daily/current.log | grep "Subject Here"

However, grep only searches on a line-by-line basis and will only match if a given line contains both of those terms. Different indicators like Senders, Recipients, Subjects etc. are often not contained on the same line.