PatrolCli - Part 1: Introduction

PatrolCli is a command-line program based on PATROL API and it runs on all PATROL agent platforms to provide access to the functions of a PATROL agent.

Unlike ProactiveNet Cli that you use to manage ProactiveNet every day, PatrolCli is not used in a standard PATROL/BPPM operation environment.  That is why many people are not even aware of its existence.  However PatrolCli is a powerful utility.  Understanding when and how to use it can save you time, lower your operations cost, and make your life a lot easier.

You can use PatrolCli to:

1. Access PATROL agents on all platforms.
2. Read PATROL objects from a PATROL agent.
3. Operate on PATROL events.
4. Execute PSL scripts.

In the next few posts, we will go through some examples on how to take advantage of PatrolCli.

PatrolCli is an executable program located at $PATROL_HOME/bin/ directory.  It is installed automatically with PATROL agent installation. You can use PatrolCli to access another PATROL agent in your network without a PATROL console.

Before you start PatrolCli, you need to set up your environment.  On Windows, you need to set up environment variable %PATROL_HOME%.  On UNIX, run '. ./patrolrc.sh' for korn shell or 'source .patrolrc' for C shell.

Here is an example on how to run PatrolCli interactively:

Myserver> PatrolCli
PCli% open RemoteServer 3181

Username: patrol

Password:

PCli% help

Here 'RemoteServer' is the host name of another server with PATROL agent running on port 3181.  When you type 'help' in PatrolCli, it will display all valid commands you can use in PatrolCli.  They include user, connect, open, list, listvars, get, getvar, event, echo, execpsl, help, source, ping, patrol, !!, exit, quit. '!!' command allows you to repeat the last command.  'exit' command will exit PatrolCli.  For each command, you can get more information by typing help followed by that command.  For example, 'help get' will give you more information on get command.

Here is an example on how you can use 'get' command in PatrolCli to display a parameter value (99.904922) and state (OK) without a PATROL console:

PCli% get /NT_CACHE/NT_CACHE/CACcachCopyReadHitsPercent
Mon Aug 25 23:42:58 2014
-1
0
99.904922
OK
3
0
0
100

PATROL Agent 9.5 - Part 7: How to configure a PATROL recovery action to run as a different user?

The information included in this post applies to all versions of PATROL agent, not just version 9.5. 

By default, if you configure a PATROL recovery action to run when a parameter is in ALARM or WARNING state, the recovery action script will be executed by PATROL default account.  What if you want to execute some commands within the recovery action script as a different account?  For example, what if you want to start SQL*Net listener within your recovery action as account 'oracle' instead of 'patrol'?

There is a PSL built-in function called execute.  Its format looks like this:

execute(type, text, [instance], [username], [password]);

For example, you can add the following PSL statement in your recovery action to start the SQL*Net listener automatically when the SQL*Net listener status parameter is in ALARM state:

execute("OS", "lsnrctl start", "", "oracle", "password");

where "password" can be either a clear text or an encrypted password.

To protect the password, you can encrypt it offline.  Then save the encrypted password in a pconfig variable.  Prior to call execute() command, you simply use command get ([variable_name]) to read the password out.

For example, if you have saved your oracle encrypted password in a pconfig variable called "/ORACLE/Encrypted_Password", you can include the following commands in your recovery action PSL script to automatically start SQL*Net when it is down.

pass = get("/ORACLE/Encrypted_Password");
execute("OS", "lsnrctl start", "", "oracle", pass); 

The reason you can use get("/ORACLE/Encrypted_Password") instead pconfig(GET,  "/ORACLE/Encrypted_Password") is because all pconfig variables are loaded into agent name space during PATROL agent start-up.

PATROL Agent 9.5 - Part 6: Do I still need PATROL Event Management KM?

You have heard from BMC for a while that PATROL Event Management KM is being phased out.  You wonder if you still need to run PATROL Event Management KM with PATROL Agent 9.5.

What does PATROL Event Management KM do?  Here are five features offered by PATROL Event Management KM:

1) Check if a server is up and if PATROL agent is up through its AS_AVAILABILITY KM;
2) Manage event generation and format through PATROL console or pconfig variable editing with or without PATROL Notification Server;
3) Set parameter thresholds through PATROL console or pconfig variable editing;
4) Turn off certain parameters through PATROL console or pconfig variable editing;
5) Add recovery actions through PATROL console or pconfig variable editing.

In PATROL agent version 9.5, the above feature 1) is replaced by automatically generated  MC_CELL_HEARTBEAT_FAILURE and MC_CELL_HEARTBEAT_ON events in BPPM cell.  When a server or PATROL agent goes down, you will automatically receive a MC_CELL_HEARTBEAT_FAILURE event.  You no longer need AS_AVAILABILITY KM.

In PATROL agent version 9.5, the above feature 2) is replaced by a set of pconfig variables starting with /EventSetup and /EventRules.  These pconfig variables provide many different ways to control and format the events from PATROL agents.  You no longer need PATROL Notification Server or use PATROL Event Management KM to control and format your events.

In PATROL agent version 9.5, the above feature 3) is replace by "Agent Threshold Configuration" section of CMA (Central Monitoring Administration).  You no longer need to use PATROL Event Management KM to set parameter thresholds.

In PATROL agent version 9.5, the above feature 4) is partially replaced by "Monitoring Configuration" of CMA when you want to turn off the data collection for the entire monitoring type (application class).  However if you only want to turn off certain parameters in an application class, you still need to use PATROL Event Management KM.

In PATROL agent version 9.5, the above feature 5) still cannot be replaced by anything else.  If you want to add a recovery action in PATROL agent without modifying the KM source code, you still need to use PATROL Event Management KM.

In summary, whether or not you can drop PATROL Event Management KM from your PATROL agent 9.5 configuration depends on if you need to turn off individual parameters and if you need to add your own recovery actions.

PATROL Agent 9.5 - Part 5: Data and event filtering

In BPPM 9.5, you can add 'Filtering Configuration' in a monitoring policy using CMA (Central Monitoring Administration).  Would this feature replace the old way you used to configure data and event filtering?

The short answer is no though you can use CMA to enhance the old way as described at the end of this post.  The 'Filtering Configuration' section in a CMA policy controls if data and events will be streamed to BPPM server or not. There are four options:

• No Filtering - the default (send all data and events)
• Filter out data (send no data, send all events)
• Filter out events (send no events, send all data)
• Filter out data and events (send no data, send no events) 

Even you select 'Filter out data and events', all data and events will still be sent to BPPM server on demand. They are just not sent as streamed.

Then how to stop sending data and events to BPPM server?  Let's take a closer look.

1) Filtering on parameter data

For data you don't need, you don't want the data being collected by PATROL KM at the first place so that you can reduce PATROL agent footprint and resource consumption.  Since CMA only allows you to control at monitoring type (application class) level, you still need to use Event Management KM to control at parameter level.  For example, if you don't want to see parameter CACcachCopyReadsPerSec, your pconfig variable settings would look like this:

"/AS/EVENTSPRING/PARAM_SETTINGS/THRESHOLDS/NT_CACHC/NT_CACHE/CACcachCopyReadsPerSec" = { REPLACE = "0,0 0 0 0 0 0,0 0 0 0 0 0,0 0 0 0 0 0" }

2) Filtering on event forwarding

For events you are not interested such as information and login events, you don't want those events being forwarded to a BPPM integration service or cell. You need to create some pconfig variable settings to activate and deactivate event types.  For example, if you only want to forward type 11 events (value of parameter is in alarm zone) and Notify Events, your pconfig variable settings would look like this:

/EventSetup/Format/BiiP3/catalogs/0/active = { REPLACE = "0"},
/EventSetup/Format/BiiP3/catalogs/0/types/11/active = { REPLACE = "1"},
/EventSetup/Format/BiiP3/catalogs/0/types/11/KM/AS_EVENTSPRING/active = { REPLACE = "0"},
/EventSetup/Format/BiiP3/catalogs/AS_EVENTSPRING/types/NOTIFY_EVENT/active = { REPLACE = "1"}

In the old way, you would save the pconfig settings for the above two cases in PCM (PATROL Configuration Manager) and push them manually to each PATROL agent they apply.  With CMA of BPPM 9.5, you can continue using PCM, or you can save them in 'Configuration Variables' section of a CMA policy so that they will automatically apply to all PATROL agents with matching tags.

PATROL Agent 9.5 - Part 4: How to include annotated data as part of the event?

The information included in this post actually applies to all versions of PATROL agent, not just version 9.5.  

Many PATROL KMs save important information to an annotated data point when changing a parameter status to ALARM or WARNING.  By default, this annotated data is not included in the event sent to BPPM cell.  What if you want to include annotated data in the event sent to BPPM cell so that it will be included in the notification email and incident ticket?

To achieve that, you will need to write a recovery action at PATROL agent.  Although PATROL Event Management KM is no longer required for PATROL agent v 9.5 from architecture point of view, to avoid modifying the original KM code, I still suggest to use PATROL Event Management KM so that you can add a recovery action by simply deploying some pconfig variable settings.

Here is an example of recovery action to include annotated data.  In this example, MS_HW_ENCLOSURE KM is used but you can apply the same concept to other KMs.  MS_HW_ENCLOSURE KM has a parameter named 'Status'.  When a blade goes down, this parameter's status becomes ALARM and an annotated data point is generated.

PATROL_CONFIG

"/AS/EVENTSPRING/MS_HW_ENCLOSURE/__ANYINST__/Status/arsAction" = { REPLACE = "6" },

"/AS/EVENTSPRING/MS_HW_ENCLOSURE/__ANYINST__/Status/arsCmdType" = { REPLACE = "PSL"},

"/AS/EVENTSPRING/MS_HW_ENCLOSURE/__ANYINST__/Status/arsCommand" = { REPLACE = "C:\\BMC\\Patrol3\\lib\\psl\\Annotate_Recovery.psl" }

Then write a PSL script C:\BMC\Patrol3\lib\psl\Annotate_Recovery.psl as follows:

sleep(1);

status = get("/MS_HW_ENCLOSURE/".__instance__."/Status/status");
txt = annotate_get("/MS_HW_ENCLOSURE/".__instance__."/Status");
txt = nthargf(txt, "2-", "\\");
origin = "MS_HW_ENCLOSURE.".__instance__.".Status_Annotate";
event_trigger2(origin, "STD", "41", status, "4", txt);

A few things to notice:

1) PATROL notification server is not required to run the above recovery action. 

2) The sleep(1) in the first line is to include a delay so that the KM has enough time to finish writing to annotated data before annotate_get() is called.

3) The nthgargf() call at the 4th line is to filter out the leading %Text %Infobox string included in every annotated data. 

4) I put 'Status_Annotate' as the last part in event origin so that this event's mc_parameter slot would become 'Status_Annotate' in the cell.  You can use any string you like.

5) The annotated data will be displayed in the event's mc_parameter_value slot.

PATROL Agent 9.5 - Part 3: Do I still need PATROL console with BPPM 9.5?

In BPPM 9.5, many features in PATROL console have been added to BPPM operations console.  I have heard this question over and over again: Do I still need PATROL console?

The answer is that you should still install PATROL console but you may not need to use it all the time and on all PATROL agents.

Let's take a look on what you use PATROL console for.

1) To develop PATROL custom knowledge modules:

Since there is no other way to develop a PATROL custom knowledge modules in BPPM 9.5, you still need to use PATROL classical console.

2) To configure PATROL knowledge modules:

With the newly available menu commands and CMA (Central Monitoring Administration) options, you can configure many PATROL knowledge modules in BPPM 9.5 operations console and CMA.  However for some PATROL knowledge modules, trying to configure them using BPPM 9.5 operations console is still very difficult or impossible.  Some features available in PATROL Central console, such as disabling parameter data collection, are still not available in BPPM operations console except by directly typing the pconfig variable values in CMA.

To avoid frustration, it is recommended to configure non-trivial PATROL knowledge modules using either PATROL Central console or PATROL classical console.  But you only need to configure one PATROL agent for each different configuration.  Carefully save all the related pconfig variables after each different configuration for each PATROL KM.  You may need to import some of them into either PCM (PATROL Configuration Manager) or CMA depending on which one you are using for mass configuration deployment. 

3) To view collected data:

You should be able to view data collected by all PATROL agents on BPPM 9.5 operations console.  For data that you configured as streamed data, you should see them displayed in real-time in BPPM 9.5 operations console as they are stored in BPPM database.  For data that you configured as non-streamed data, you should still be able to see them as BPPM 9.5 operations console retrieves them from PATROL agents on demand.  However non-streamed data may be displayed in BPPM 9.5 operations console with a delay.

4) Trouble shooting:

In theory, you should be able to perform most of the trouble shooting by using BPPM 9.5 operations console.  However due to possible delay of non-streamed data and the fact that PATROL Central console Windows edition is many times faster than BPPM 9.5 operations console (a web GUI), you would most likely prefer to use PATROL console for trouble shooting.  If you need to verify/change some complicated KM configuration, you would have to use PATROL console.

In summary, with BPPM 9.5, you would most likely use your PATROL classical console as developer console, your PATROL Central console as admin console, and your BPPM console as operations console.

PATROL Agent 9.5 - Part 2: When PATROL agent hangs

In the last post, we discussed how to detect when PATROL agent is unreachable including network down, server down, and PATROL agent down.  One of my readers asked me if the same method can be used to detect if PATROL agent hangs.

When PATROL agent hangs, PATROL agent process is still up with a heartbeat.  Therefore the method we discussed in the last post won't be able to tell if PATROL agent hangs.  When PATROL agent hangs, it just stops collecting data.

The best way to detect when PATROL agent hangs is to write your own script to check when the last data collection happened.  Pick up a common parameter with frequent data collection, e.g., /NT_CPU/CPU__Total/CPUprcrProcessorTimePercent on Windows.  By default, CPUprcrProcessorTimePercent is collected every minute.  When you query /NT_CPU/CPU__Total/CPUprcrProcessorTimePercent/time, you will receive the last data collection time stamp as Epoch time.  Compare this time with the current time stamp, you will know how long it has been since CPU data was last collected.  If it has been a while since CPU data was last collected (e.g. longer than 10 minutes), you can reasonably conclude that PATROL agent hangs.

You can use PatrolCli to query /NT_CPU/CPU__Total/CPUprcrProcessorTimePercent/time.  PatrolCli is an executable program installed automatically with PATROL agent under $PATROL_HOME/bin directory.  It allows you to connect to any PATROL agent in your network with the right password and permission.  And it allows you to run a subset of PSL commands from command line.  Type 'PatrolCli' from the command line and use 'help' command to see all the commands it supports.

Because you are trying to detect if PATROL agent hangs, it would be the best to use a mechanism outside of PATROL framework to schedule your script and send out the alerts.  For example, you can use cron on UNIX or scheduled task on Windows to run your script periodically.  When you detect PATORL agent hangs, you can send alert emails or use msend command to directly report your alerts to BPPM/BEM cell.

I will be out of country and won't be able to post any blogs for the next two weeks.  This blog will resume on July 21, 2014.  Have a great summer!