Event Integration From 3rd-party Monitoring Tools - Part 2: Msend API installation

The easiest and most reliable way to integrate events from 3rd-party monitoring software to BPPM is by using msend API.  Msend API is an executable program available as part of BPPM/BEM cell installation.  It is available on operating systems that BPPM/BEM cell supports including Windows, Linux, Solaris, AIX, and HP-UX.

In order to use msend API, the 3rd-party monitoring software must provide an option to execute an OS script when an alert is raised. Over 90% of the 3rd-party monitoring software out there do provide this option.

This post covers the details on msend API packaging and installation.  The next post will cover the best practice to execute OS script.

First, you need to put a msend API package together since BMC doesn't provide a separate msend API installation package.  Decide which operating system you need to install msend API.  This is the operating system that your 3rd-party monitoring software is running on.  

If you already have a BPPM/BEM cell installation for that operating system, you can just copy from it.  If you don't, you can either install an instance of BPPM/BEM cell on the operating system you need or you can simply ask BMC support to send you an msend executable for the operating system you need.

Copy the following files from your existing BPPM/BEM cell installation.  Replace msend (on UNIX) or msend.exe (on Windows) for a different operating system.  This is the only file you need to replace as all other files are ASCII files.  Your installation package should contain the following directory structure:

bin\

   msend.exe (for Windows) or msend (for UNIX)

etc\

   mcell.dir 

   mclient.conf 

   mclient.trace

etc\locale\

      mclient.load 

      imgwmsg.properties 

      langmsg.properties 

      mccompmsg.properties 

      mcellmsg.properties 

      mclimsg.properties 

      mcmdmsg.properties

Simply zip or tar the above directory structure.  When you unzip or untar your installation package on the server where 3rd-party monitoring software runs, your msend API is ready to use.

Event Integration From 3rd-party Monitoring Tools - Part 1: Introduction

Although the capability of integrating performance data from 3rd-party monitoring software into BPPM is still very limited, the capability of integrating events (alerts) from 3rd-party monitoring software into BPPM is extremely powerful and flexible.

Why do you want to integrate events from 3rd-party monitoring software into BPPM?

First of all, if you have a NOC team monitoring the alerts sent from different monitoring tools, it would make sense for the NOC team to watch one computer screen instead of multiple computer screens.

Second, if you use Remedy to create incident tickets, BPPM has a fast and reliable native integration with Remedy ticket system. And the integration license is included in BPPM license.  If the 3rd-party monitoring tool doesn't have its own integration with Remedy, integrating its events into BPPM would instantly add that capability.  If the 3rd-party monitoring tool requires you to buy a separate license for Remedy integration, integrating its events into BPPM would be a great cost saving.

Third, by having events from different monitoring tools in the same event database, you can develop correlation rules to create tickets only on root cause components thus to eliminate large number of unnecessary Remedy tickets.  For example, when a switch down event from network monitor caused 100 server down events in server monitor, you only need to create a Remedy ticket for switch down event.

BMC provides three general-purpose methods to integrate events from 3rd-party monitoring software into BPPM: msend API, SNMP trap adapter, and iiSDK.  I have personally used all three. I highly recommend msend API as the easiest and most reliable integration method.  So far over 90% of my integration work is based on msend API.

In addition, if your 3rd-party monitoring software cannot call msend API but can send emails, you can use email as the 4th method for event integration.

In the next few posts, I will discuss each method in detail.

bii4P3 vs bii4P7 - Part 2: How to set up high availability for bii4P

In last post, I mentioned that neither bii4P3 nor bii4P7 has built-in high availability.  If system monitoring is a critical component of your overall IT infrastructure, without high availability, bii4P can become a single point of failure.

For example, one of my previous clients manages IT infrastructure for a group of hospitals. In their environment, every single component is required to set up as high availability. Missing a critical "application down" alert could mean the difference between life and death for a hospital.

There are two methods that you can set up your own high availability for bii4P whether you choose bii4P3 or bii4P7.  However as I explained in my last post, bii4P3 is a more stable solution.

The first method is a simple redundancy.  For each bii4P instance you configured, you configure another identical bii4P instance running on another server.  You run both instances at the same time.  In BPPM/BEM cell, you use de-dup rule to drop the duplicated events.  

The advantage of the first method is simplicity.  The only MRL rule you need to pay attention to is the de-dup rule for PATROL events.  If you read BMC's out-of-box de-dup rule for PATROL events closely, you will find that you need to make some modification to it. Otherwise some non-duplicated events could be de-duped incorrectly - you could miss a critical alert because of that.  The disadvantage is that you have to double the network traffic.

The second method is to write a MRL rule to coordinate between two instances of bii4P.  For each bii4P instance you configured, you still configure another identical bii4P instance running on another server.  But you only run one instance at a time.  When the first instance went down, there will be a 'bii4P down' event shown up in BPPM/BEM cell to activate the MRL rule.  The rule will start the second instance of bii4P and notify BPPM administrator.  

The advantage of the second method is that it works as an application-level failover without increasing network traffic.  The disadvantage is that you do need to have experience in MRL programming and OS scripting.  I found that having each instance of bii4P share the same server as the cell simplifies the OS scripting work.

bii4P3 vs bii4P7 - Part 1: Why should you choose bii4P3

bii4P3 and bii4P7 are PATROL event adapters to connect PATROL agent older than version 9.0 with BPPM/BEM cell.  If all your PATROL agents are version 9.x and above, you don't need bii4P3 nor bii4P7 since PATROL agent v9.x can directly send events to BPPM/BEM cell. However both bii4P3 and bii4P7 still work for PATROL agent v9.x if you want to use them.

What is the difference between bii4P3 and bii4P7?  bii4P3 is a light-weight adapter at one end connecting to each PATROL agent directly and at the other end connecting to BPPM/BEM cell.  bii4P7 uses common connect library at one end connecting to PATROL console server in PATROL 7 architecture and at the other end connecting to BPPM/BEM cell.  bii4P3 was released a couple of years earlier than bii4P7.

Because PATROL console server was initially designed as a GUI component only with no built-in high availability, it was not meant to be robust enough to be used as an infrastructure integration component. When bii4P7 connects with PATROL console server, it competes with all PATROL Central users for available resources.  If not enough resource is available, bii4P7 would crash or hang, thus no event would be sent to BPPM/BEM cell.  Thus bii4P7 could become the single point of failure in your entire system monitoring architecture.

bii4P3 directly communicates with PATROL agents and completely bypasses PATROL console server or any PATROL console component.  Because it is light weight, it can be installed on any supported server (Linux, Solaris, and Windows).  The recommended capacity is 100 PATROL agents per bii4P3 instance. But you can run multiple instances on one server.

If you use PATROL Notification Server, you can connect bii4P3 to it.  However PATROL Notification Server is not required to use bii4P3 as you can configure multiple bii4P3 instances on one server or multiple servers.

I have replaced bii4P7 with bii4P3 at two of my previous client sites to repair their unstable environment.  I am currently at another client who desperately needs to replace its bii4P7 with bii4P3 too.  Their bii4P7 is running on an AIX server.  I have seen the core file produced by bii4P7 almost at daily basis.  When that happens, bii4P7 needs to be started manually. About half the time, bii4P7 appears to be started with process shown as the output of 'ps -ef' command, but no event was received by BPPM/BEM cell.  In this case bii4P7 hangs and needs to be stopped and started again.  Sometimes PATROL console server needs to be restarted in order to start bii4P7.

Neither bii4P3 nor bii4P7 has built-in high availability. In the next post, I will talk about how to configure your own high availability for bii4P3.