Event Integration From 3rd-party Monitoring Tools - Part 5: Msend API mapping and conversion

In your 3rd-party monitoring software, each event (also called alert) contains many different fields (also called attributes).  You need to map those field names to BPPM cell slot names in order to pass those field values into BPPM cell. 

First, you need to decide what BPPM cell class you want to map to.  You can map to base class EVENT or an existing subclass of EVENT such as PATROL_EV.  Or, you can define your own class.  I personally prefer to define a specific class for each 3rd-party monitoring software regardless if I need to add extra slots to it or not.  For example, I have defined SCOM_EV for events from Microsoft SCOM and NAGIOS_EV for events from Open Source tool Nagios.

Second, you need to decide what slots in BPPM cell need to be mapped.  Each event in BPPM cell has more than 75 slots.  Obviously you don't need map all 75 slots.  If you will send emails for some events, the slots referenced in your email will need to be mapped.  If you will create incident tickets from some events, the slots that map to your ticketing system will need to be mapped from your 3rd-party monitoring software.  If you have a NOC team that watches BPPM GUI to help create and escalate tickets, the slots your NOC team can see will need to be mapped. Commonly mapped BPPM cell slots include mc_host, mc_object_class, mc_object, mc_tool_class, mc_tool_key, mc_parameter, mc_parameter_value, severity, mc_priority, and msg.

Third, you need to decide how to map. For most of BPPM cell slots, you can map directly.  For example, you can use mc_host=${HOSTNAME} to map HOSTNAME field from your 3rd-party monitoring software directly to BPPM cell slot mc_host.  However for some slots in BPPM cell, especially the ones in enumerable type, you may need to perform value conversion unless BPPM cell and your 3rd-party monitoring software use exactly the same spelling for all their values. For example, if your 3rd-party monitoring software uses high, medium, and low as the value of event priority, but BPPM cell uses PRIORITY_1 to PRIORITY_5 as the values of events priority, you would need to convert the values in your 3rd-party monitoring software to the values in BPPM cell.

You can perform conversion in your OS script. Or you can map the conversion-required fields to other string-typed BPPM slots 'as is'.  Then you can create a MRL rule in BPPM cell to perform the conversion.  In the above example, you can create a new BPPM cell slot called priority_string to map priority field from your 3rd-party monitoring software.  Then in your MRL rule, you can convert value in priority_string slot to mc_priority slot.  By performing conversion in BPPM cell, you can take advantage of dynamic tables to make value conversion easy to code and easy to maintain.

Event Integration From 3rd-party Monitoring Tools - Part 4: Msend API execution

After installing msend API package and successfully testing the communication between your 3rd-party monitoring software server and your BPPM/BEM cell, you are ready to call msend from your 3rd-party monitoring software.

In your 3rd-party monitoring software, first locate where you can configure an OS script when an alert is raised.  Usually you will need to specify a condition (such as a specific severity or a specific component) and the name of an OS script.

Although you can directly call msend by using 'msend' as your OS script name, I recommend that you create a separate OS script (e.g. call_msend).  Your 3rd-party monitoring software will call this OS script when an alert is raised.  Then this OS script will call msend.  This way you can set the environment to execute msend inside your OS script without having to merge msend environment variables into your 3rd-party monitoring software environment - which would often require you to restart your 3rd-party monitoring software.  In addition, you can unit test your OS script separately without having to wait for an alert to happen.

Second, decide which system variables (also called micros in some 3rd-party monitoring software) you would like to send to BPPM/BEM cell. These system variables contain the values of hostname, component name, parameter name, parameter value, etc. from the raised alert. The values of these system variables will be used when BPPM/BEM cell sends a notification email or creates an incident ticket.  You will configure these system variables as arguments to your OS script.

Third, construct your OS script. In the script, set up environment to execute msend, use buffer mode (-j) for guaranteed delivery, and decide to send events from command line or a text file. Examples of sending events in buffer mode, sending events from command line, and sending events from a text file were listed in the previous post "Event Integration From 3rd-party Monitoring Tools - Part 3: Msend API Configuration".

Sending events from command line is a little faster than sending events from a text file because no file I/O is involved, but sending events from a text file is easier for trouble shooting since you can keep each text file as a sender's log.  If you do choose to send events from a text file, make sure that you always use a unique file name for each event.  You can delete the text file immediately after sending the event or manually delete them later when you no longer need them for trouble shooting purpose.

In the next post, we will discuss mapping and conversion when using msend API.

Event Integration From 3rd-party Monitoring Tools - Part 3: Msend API configuration

Before you install msend API package, on the server where your 3rd-party monitoring software runs, decide your installation directory <Install_Dir> and define the following environment variable:

$MCELL_HOME=<Install_Dir>/BMC/Server (for UNIX) or

%MCELL_HOME%=<Install_Dir>\BMC\Server (for Windows)

Then add $MCELL_HOME/bin or %MCELL_HOME%\bin to the environment variable PATH.

Now you can untar or unzip your msend API package under %MCELL_HOME% directory.  After all files are installed, open %MCELL_HOME%\etc\mcell.dir and verify all cells are defined correctly.  Then create a directory called "buffer" under %MCELL_HOME%.

For UNIX, verify that $MCELL_HOME/bin/msend has its execution bit turned on.  Run 'chmod +x msend' if needed. In addition, in order to run msend API in buffer mode, you need to create a file called /etc/itm/.reg and make /etc/itm/.reg writable to the account executing msend API.  You need root privilege to create /etc/itm/.reg.

Before calling msend API from your 3rd-party monitoring software, I highly recommend testing the communication by using the following simple test cases:

1. Send an event from command line: 

    msend -n <cell_name> -a EVENT -m "Test from command line"

2. Send an event from command line in buffer mode:

    msend -n <cell_name> -j <Install_Dir>\BMC\server\buffer -a EVENT -m "Test from command line with buffer"

3. Send an event from a text file:

    msend -n <cell_name> myevent1.txt

where myevent1.txt should contain these lines:

EVENT;

       severity=CRITICAL;

       msg="Test from text file";

END

4. Send an event from a text file in buffer mode:

    msend -n <cell_name> -j <Install_Dir>\BMC\server\buffer myevent2.txt

where myevent2.txt should contain these lines:

EVENT;

       severity=CRITICAL;

       msg="Test from text file with buffer";

END

   

In each of the above four cases, verify that an event is present in BPPM/BEM GUI and no error message is displayed from msend command. If you are running msend in buffer mode, also verify that 'msend' process is still running up to 10 minutes after you call msend command.

We will cover msend API execution in the next post.

Event Integration From 3rd-party Monitoring Tools - Part 2: Msend API installation

The easiest and most reliable way to integrate events from 3rd-party monitoring software to BPPM is by using msend API.  Msend API is an executable program available as part of BPPM/BEM cell installation.  It is available on operating systems that BPPM/BEM cell supports including Windows, Linux, Solaris, AIX, and HP-UX.

In order to use msend API, the 3rd-party monitoring software must provide an option to execute an OS script when an alert is raised. Over 90% of the 3rd-party monitoring software out there do provide this option.

This post covers the details on msend API packaging and installation.  The next post will cover the best practice to execute OS script.

First, you need to put a msend API package together since BMC doesn't provide a separate msend API installation package.  Decide which operating system you need to install msend API.  This is the operating system that your 3rd-party monitoring software is running on.  

If you already have a BPPM/BEM cell installation for that operating system, you can just copy from it.  If you don't, you can either install an instance of BPPM/BEM cell on the operating system you need or you can simply ask BMC support to send you an msend executable for the operating system you need.

Copy the following files from your existing BPPM/BEM cell installation.  Replace msend (on UNIX) or msend.exe (on Windows) for a different operating system.  This is the only file you need to replace as all other files are ASCII files.  Your installation package should contain the following directory structure:

bin\

   msend.exe (for Windows) or msend (for UNIX)

etc\

   mcell.dir 

   mclient.conf 

   mclient.trace

etc\locale\

      mclient.load 

      imgwmsg.properties 

      langmsg.properties 

      mccompmsg.properties 

      mcellmsg.properties 

      mclimsg.properties 

      mcmdmsg.properties

Simply zip or tar the above directory structure.  When you unzip or untar your installation package on the server where 3rd-party monitoring software runs, your msend API is ready to use.

Event Integration From 3rd-party Monitoring Tools - Part 1: Introduction

Although the capability of integrating performance data from 3rd-party monitoring software into BPPM is still very limited, the capability of integrating events (alerts) from 3rd-party monitoring software into BPPM is extremely powerful and flexible.

Why do you want to integrate events from 3rd-party monitoring software into BPPM?

First of all, if you have a NOC team monitoring the alerts sent from different monitoring tools, it would make sense for the NOC team to watch one computer screen instead of multiple computer screens.

Second, if you use Remedy to create incident tickets, BPPM has a fast and reliable native integration with Remedy ticket system. And the integration license is included in BPPM license.  If the 3rd-party monitoring tool doesn't have its own integration with Remedy, integrating its events into BPPM would instantly add that capability.  If the 3rd-party monitoring tool requires you to buy a separate license for Remedy integration, integrating its events into BPPM would be a great cost saving.

Third, by having events from different monitoring tools in the same event database, you can develop correlation rules to create tickets only on root cause components thus to eliminate large number of unnecessary Remedy tickets.  For example, when a switch down event from network monitor caused 100 server down events in server monitor, you only need to create a Remedy ticket for switch down event.

BMC provides three general-purpose methods to integrate events from 3rd-party monitoring software into BPPM: msend API, SNMP trap adapter, and iiSDK.  I have personally used all three. I highly recommend msend API as the easiest and most reliable integration method.  So far over 90% of my integration work is based on msend API.

In addition, if your 3rd-party monitoring software cannot call msend API but can send emails, you can use email as the 4th method for event integration.

In the next few posts, I will discuss each method in detail.

bii4P3 vs bii4P7 - Part 2: How to set up high availability for bii4P

In last post, I mentioned that neither bii4P3 nor bii4P7 has built-in high availability.  If system monitoring is a critical component of your overall IT infrastructure, without high availability, bii4P can become a single point of failure.

For example, one of my previous clients manages IT infrastructure for a group of hospitals. In their environment, every single component is required to set up as high availability. Missing a critical "application down" alert could mean the difference between life and death for a hospital.

There are two methods that you can set up your own high availability for bii4P whether you choose bii4P3 or bii4P7.  However as I explained in my last post, bii4P3 is a more stable solution.

The first method is a simple redundancy.  For each bii4P instance you configured, you configure another identical bii4P instance running on another server.  You run both instances at the same time.  In BPPM/BEM cell, you use de-dup rule to drop the duplicated events.  

The advantage of the first method is simplicity.  The only MRL rule you need to pay attention to is the de-dup rule for PATROL events.  If you read BMC's out-of-box de-dup rule for PATROL events closely, you will find that you need to make some modification to it. Otherwise some non-duplicated events could be de-duped incorrectly - you could miss a critical alert because of that.  The disadvantage is that you have to double the network traffic.

The second method is to write a MRL rule to coordinate between two instances of bii4P.  For each bii4P instance you configured, you still configure another identical bii4P instance running on another server.  But you only run one instance at a time.  When the first instance went down, there will be a 'bii4P down' event shown up in BPPM/BEM cell to activate the MRL rule.  The rule will start the second instance of bii4P and notify BPPM administrator.  

The advantage of the second method is that it works as an application-level failover without increasing network traffic.  The disadvantage is that you do need to have experience in MRL programming and OS scripting.  I found that having each instance of bii4P share the same server as the cell simplifies the OS scripting work.

bii4P3 vs bii4P7 - Part 1: Why should you choose bii4P3

bii4P3 and bii4P7 are PATROL event adapters to connect PATROL agent older than version 9.0 with BPPM/BEM cell.  If all your PATROL agents are version 9.x and above, you don't need bii4P3 nor bii4P7 since PATROL agent v9.x can directly send events to BPPM/BEM cell. However both bii4P3 and bii4P7 still work for PATROL agent v9.x if you want to use them.

What is the difference between bii4P3 and bii4P7?  bii4P3 is a light-weight adapter at one end connecting to each PATROL agent directly and at the other end connecting to BPPM/BEM cell.  bii4P7 uses common connect library at one end connecting to PATROL console server in PATROL 7 architecture and at the other end connecting to BPPM/BEM cell.  bii4P3 was released a couple of years earlier than bii4P7.

Because PATROL console server was initially designed as a GUI component only with no built-in high availability, it was not meant to be robust enough to be used as an infrastructure integration component. When bii4P7 connects with PATROL console server, it competes with all PATROL Central users for available resources.  If not enough resource is available, bii4P7 would crash or hang, thus no event would be sent to BPPM/BEM cell.  Thus bii4P7 could become the single point of failure in your entire system monitoring architecture.

bii4P3 directly communicates with PATROL agents and completely bypasses PATROL console server or any PATROL console component.  Because it is light weight, it can be installed on any supported server (Linux, Solaris, and Windows).  The recommended capacity is 100 PATROL agents per bii4P3 instance. But you can run multiple instances on one server.

If you use PATROL Notification Server, you can connect bii4P3 to it.  However PATROL Notification Server is not required to use bii4P3 as you can configure multiple bii4P3 instances on one server or multiple servers.

I have replaced bii4P7 with bii4P3 at two of my previous client sites to repair their unstable environment.  I am currently at another client who desperately needs to replace its bii4P7 with bii4P3 too.  Their bii4P7 is running on an AIX server.  I have seen the core file produced by bii4P7 almost at daily basis.  When that happens, bii4P7 needs to be started manually. About half the time, bii4P7 appears to be started with process shown as the output of 'ps -ef' command, but no event was received by BPPM/BEM cell.  In this case bii4P7 hangs and needs to be stopped and started again.  Sometimes PATROL console server needs to be restarted in order to start bii4P7.

Neither bii4P3 nor bii4P7 has built-in high availability. In the next post, I will talk about how to configure your own high availability for bii4P3.