PATROL LOG KM Examples - Part 3: A simple case of multiple-line search

In the last two PATROL LOG KM posts, I have discussed two different ways to send out alerts. In those alerts, each matched log entry contains one single line from the log file.  What if you want each matched log entry to contain more than one line?  This happens when some critical information is actually contained in the lines before or after the line that matches the search string pattern.  Including those additional lines in your alert emails or trouble tickets would definitely help to speed up the troubleshooting process.

If the additional lines you want to include are after the line that matches the search string pattern, the solution is simple.  For example, if you would like to have the following two lines included in your matched log entry:

031605: Error: Disc Full
/hd001 mounted as /opt

You can use 'Disc Full' as your search string pattern.  To make the matched log entry contain one additional line after the line that matched the search string pattern, you simply put '2' in 'Number of Lines in Log Entry' field in LOG KM instance configuration screen.  (Please see the location of this field from the LOG KM instance configuration screen displayed in 'PATROL LOG KM Examples - Part 2' post.)  And you can configure the rest of LOG KM as usual.  You can send one alert per polling cycle as described in 'PATROL LOG KM Examples - Part 1' post or send one alert per matched log entry as described in 'PATROL LOG KM Examples - Part 2' post.

However, if the additional lines you want to include are before the line that matches the search string pattern, the solution is not so simple.  For example, if you would like to have the following two lines included in your matched log entry:

root 21292 c Mon Oct 28 08:00:00 2013
! Your password will expire in 3 days.

Here you would need to use some strings from the second line as your search string pattern because nothing from the first line is unique enough as a search pattern.  Then how can we include information from the line before the line that matches the search string pattern?  In the next post, I will discuss a solution to this example by using an advanced feature of PATROL LOG KM called 'Multiline Search'.  Stay tuned.

PATROL LOG KM Examples - Part 2: Sending one alert per matched log entry

In the last post "PATROL LOG KM Examples - Part 1", I discussed how to configure PATROL LOG KM to send one alert per polling cycle regardless how many matched log entries were found in the polling cycle. But what if some of the matched log entries are database related alerts and need to be emailed and ticketed against database group, and some of the matched log entries are operating system related alerts and need to be emailed and ticketed against UNIX sysadmin group?

There is another way to configure PATROL LOG KM to send one alert per matched log entry. This option is lesser known, but it is more flexible than sending one alert per polling cycle because you can specify alert severity separately for each string pattern.  For example, you can specify severity ALARM for each log entry that matches string pattern "fatal", and specify severity WARNING for each log entry that matches string pattern "retry".

To send one alert per matched log entry, you need to configure "Default Settings for Search Criteria" section as shown in the following example:

"Custom Event Origin" should contain three strings separated by '.'.  The first string before '.' (%APPCLASS% in the above example) will go to mc_object_class slot in your event.  The 2nd string between two '.'s (%FILENAME% in the above example) will go to mc_object slot in your event.  The 3rd string after the '.' (%LOGICALNAME% in the above exmaple) will go to mc_parameter slot in your event.  In the above example, you will get an event with

mc_object_class='LOGMON';

mc_object='C:\BMC\Patrol3\log\PatrolAgent-Sophie-3181.errs';

mc_parameter='PATROL_AGENT_LOG';

"Custom Event Message" should contain anything you want to show in msg slot of your event.  In the above example. I put "%SEARCHID%:%1-".  If you specify your search ID as "FATAL" for your string pattern "fatal", and the log entry that matches "fatal" string pattern is "Fatal error.  Application exit.", the msg slot in your event will be:

msg='FATAL:Fatal error.  Application exit.';

This is the only screen your need to configure to let PATROL LOG KM send one alert per matched log entry.  Unlike the previous post, you don't need to do anything in pconfig or coding in PSL at all.

PATROL LOG KM Examples - Part 1: Sending one alert per polling cycle

Sorry for not posting for two weeks as I was out of country where I was not able to access this blog.

PATROL LOG KM is one of the most commonly used KMs.  By design, each LOG KM instance monitors one log file.  Two important parameters in LOG KMs are LOGErrorLvl and LOGMatchString. When a string pattern is found in the log file, LOGErrorLvl will go to alarm and the matched log entry will be saved in LOGMatchString. Since you can configure LOG KM to search for multiple string patterns in each log file, all matched log entries are saved together in one LOGMatchString parameter.

If you would like to send all matched log entries as one alert, you can use recovery action to generate an event and send to BPPM/BEM cell.  I have seen many examples that use variable '__udefvar__' in pconfig rules.  But  '__udefvar__' only works with PATROL Notification Server. If you don't use PATROL Notification Server as it is optional to use it prior to PATROL agent 9.x and there is no need to use it with PATROL agent 9.x, you can use event_trigger2() PSL call instead. Here is an example pconfig rule set and PSL code.

Pconfig rule:

"/AS/EVENTSPRING/LOGMON/__ANYINST__/LOGErrorLvl/arsAction" = { REPLACE = "6" },
"/AS/EVENTSPRING/LOGMON/__ANYINST__/LOGErrorLvl/arsCmdType" = { REPLACE = "PSL"},
"/AS/EVENTSPRING/LOGMON/__ANYINST__/LOGErrorLvl/arsCommand" = { REPLACE = "C:\\BMC\\Patrol3\\lib\\psl\\LOGKM_RecoveryAction.psl" }

LOGKM_RecoveryAction.psl code: 

sleep(1);

message = get("/LOGMON/". __instance__."/LOGMatchString/value");

inst= get("/LOGMON/". __instance__."/name");

event_trigger2(inst."/LOGMatchString","STD", "41", ALARM, 4, message);

set("/LOGMON/".__instance__."/LOGErrorLvl/value", 1);

A few things to notice here:

1. You can embed the entire PSL script into "/AS/EVENTSPRING/LOGMON/__ANYINST__/LOGErrorLvl/arsCommand" pconfig variable with some '\' to escape newlines and "()" symbols, etc.

2. The sleep statement in the first line would give PATROL agent enough time to finish writing a big block of data into LOGMatchString.

3. The set statement in the last line sets parameter LOGErrorLvl back to OK state immediately after the recovery action.  Recovery action is triggered by state change.  When another string pattern is found again in the next polling cycle, if the state of parameter LOGErrorLvl remains in ALARM state without going back to OK in between, the recovery action won't be triggered.