Event Integration From 3rd-party Monitoring Tools - Part 9: SNMP trap adapter execution and high availability

With BMC SNMP Trap Adapter, most of the work required for execution has been done out of box in mcxa executable and its configuration file etc\mcxa.conf. However, there are a few things specific about SNMP traps that we do want to pay attention to.

First, if your 3rd-party monitoring software allows, filter the SNMP traps to limit the number of traps to what you really need.  SNMP traps are in general very chatty.  Without filtering, you can expect to see large number of SNMP traps sending to your BPPM cell.

Second, pay attention to SNMP trap adapter log file. If you have large traps, the error messages in your log file will tell you to increase the value of parameter SnmpTrapLength in mcxa.conf.  The default value for SnmpTrapLength is 8192 bytes.  You need to double that value for large traps.

Third, implement high availability for SNMP trap adapter.  In general, SNMP traps are not logged, buffered, or re-tried.  You would easily miss some important SNMP traps from a short period of network outages. However since BMC doesn't provide out-of-box high availability for SNMP trap adapter, we would have to go with simple redundancy.

Install and configure two identical instances of SNMP trap adapter on two different servers. If you have an active/standby BPPM cell cluster, you can install a SNMP trap adapter on each server of the cluster.  This is simply a convenience, not a requirement.  Then configure these two SNMP trap adapter severs as two SNMP trap destinations from your 3rd-party monitoring software.

Now for each SNMP trap sent out from your 3rd-party monitoring software, you will receive two events with one event converted from each SNMP trap adapter.  You will need to write a de-dup MRL rule in your BPPM cell to drop one of the events.  Your de-dup rule should compare the original trap IDs to determine if two traps are identical.  If the trap ID is short, meaning that the same ID can be re-used fairly quickly, you will also need to compare the original timestamps as well to determine if two events are truly duplicates.

Event Integration From 3rd-party Monitoring Tools - Part 8: SNMP trap adapter configuration from command line

We discussed how to configure SNMP trap adapter using GUI in the last post. What if you can't find a server with no TOMCAT running to install SNMP Configuration Manager?  You do have an option to configure SNMP trap adapter from command line.

Method 2: From command line

In order to configure SNMP trap adapter from command line, you need to download and install open source net-snmp package from http://www.net-snmp.org/download.html

Once it is installed, define an environment variable SNMP_DIR pointing to net-snmp installation direcotry.  For example:

set SNMP_DIR=C:\SNMPAdapter\ucd-snmp-4.2.3\usr

Then copy your MIB file to %SNMP_DIR%\mibs\ directory.  Run the following command:

mib2map.pl -m ALL -M "%SNMP_DIR%\mibs;%SNMP_DIR%\share\snmp\mibs" -s %SNMP_DIR%\bin\snmptranslate.exe

Where mib2map.pl is a perl script already installed in your %MCELL_HOME%\bin\ directory as part of BMC Event Adapters.

In the current directory, you should see the following 4 files generated as the result of running the above mib2map.pl command:

mcsnmptrapdmib.dat

mcsnmptrapdmib.map

mcsnmptrapdmib.baroc

mcsnmptrapdmibe.baroc

Now you have generated the above 4 files either from GUI or command line, copy both mcsnmptrapdmib.baroc and mcsnmptrapdmibe.baroc to %MCELL_DIR%\etc\<cell_name>\kb\classes\ directory.  Add mcsnmptrapdmibe followed by mcsnmptrapdmib to .load file in %MCELL_DIR%\etc\<cell_name>\kb\classes\ directory.  The <cell_name> here needs to match the value of parameter ServerName in your mxca.conf file.

Copy both mcsnmptrapdmib.dat and mcsnmptrapdmib.map to %MCELL_DIR%\etc\ directory. Rename mcsnmptrapdmib.dat to mcsnmptrapd.dat.  Rename mcsnmptrapdmib.map to mcsnmptrapd.map.

Recompile the cell.  Restart the cell.  Stop and then start SNMP trap adapter after a couple of minutes.  Now you can configure your 3rd-party monitoring software to send SNMP traps to your SNMP trap adapter.  You should see events with class names defined in the MIB file shown up in your cell.

Event Integration From 3rd-party Monitoring Tools - Part 7: SNMP trap adapter configuration using GUI

Since SNMP trap adapter is responsible to translate a SNMP trap into an event in BAROC format, it needs some instruction to know how to translate.  This instruction is contained in the MIB file provided by the vendor of your 3rd-party monitoring software. You need to ask the administrator of your 3rd-party monitoring software to send you a copy of MIB file before you start configuring SNMP trap adapter.

The purpose of configuring SNMP trap adapter is to generate the following 4 files from your MIB file:

mcsnmptrapdmib.dat

mcsnmptrapdmib.map

mcsnmptrapdmib.baroc

mcsnmptrapdmibe.baroc

You can choose either GUI method or command line method to achieve that.  This is an one-time process and you will use the same 4 files for all your development, QA, and production environments.  You only need to go through this configuration process again if your MIB file has changed.

Method 1: Using GUI

Download SNMP Configuration Manager from BMC EPD site.  Install SNMP Configuration Manager. You will be asked to input JDK Home directory, Perl Home directory, and TOMCAT port numbers. The default TOMCAT port numbers are 8080 (listening) and 8005 (shutdown). Verify both ports are available first.

Since this is an one-time process and SNMP Configuration Manager is not needed during run time, I recommend to install SNMP Configuration Manager on a test server that has no TOMCAT running.  It is best not to install SNMP Configuration Manager on the same server where BPPM server is located to avoid TOMCAT port conflict.

Once SNMP Configuration Manager has been installed, you can access it through this URL:

http://<servername>:8080/snmpadapter/

SNMP Configuration Manager will ask you for the location of your MIB file.  As the result of publishing the MIB file, it will generate the above four files. Verify that mcsnmptrapdmib.dat and mcsnmptrapdmib.map are located in %MCELL_HOME%\etc\ directory.  Verify that mcsnmptrapdmib.baroc and mcsnmptrapdmibe.baroc are located in %MCELL_HOME%\bin\ directory.

We will discuss how to configure SNMP trap adapter using command line in the next post.

Event Integration From 3rd-party Monitoring Tools - Part 6: SNMP trap adapter installation

SNMP trap adapter is another way to integrate events from your 3rd-party monitoring software into a BPPM/BEM cell. About 90% of monitoring software out there can send SNMP traps when alerts are raised. SNMP trap adapter is a perl-based adapter provided by BMC to translate SNMP traps to events in BAROC format, a format that BPPM/BEM cell can understand.  By default, SNMP trap adapter listens on port 162.

SNMP trap adapter is part of BMC ProactiveNet Event Adapters. It is often installed on the same server where BPPM/BEM cell is installed.  However, it is not required to have SNMP trap adapter and the BPPM/BEM cell it connects to co-locate on the same server.

Although BMC talks about SNMP trap adapter all the time and barely mention msend API, I personally prefer msend API whenever possible and use SNMP trap adapter only if the 3rd-party monitoring software is not capable of executing an OS script when an alert is raised. When buffer mode '-j' is used, events sent using msend API will be guaranteed to arrive at the BPPM cell.  If the cell is down or unreachable when your 3rd-party monitoring software is calling msend API, msend will buffer the unsent events and keep trying until the cell is up.  If SNMP trap adapter is down or unreachable when your 3rd-party monitoring software is sending SNMP traps, all SNMP traps will be lost.

Before starting SNMP trap adapter, edit mcxa.conf.  Locate [Snmp] section.  If you use SNMP version 3, locate [SnmpV3] section. Remove the word 'DISABLE'.  Add the following lines to maximize SNMP trap throughput.

PollInterval     = 1

ReadsPerEngine   = 200

SnmpRcvbuf       = 262144

If ServerName is not set to your cell name under [default] section, you also need to set ServerName to your cell name under [Snmp] section.

Now you can start your SNMP trap adapter.  Check the adapter status again after a couple of minutes and verify it stays up.  If you need to go back and change mcxa.conf and restart SNMP trap adapter, wait for a couple of minutes between stop and start the adapter to give it enough time to release port 162.

We will discuss configuration, execution, mapping, and conversion in the next few posts.

Event Integration From 3rd-party Monitoring Tools - Part 5: Msend API mapping and conversion

In your 3rd-party monitoring software, each event (also called alert) contains many different fields (also called attributes).  You need to map those field names to BPPM cell slot names in order to pass those field values into BPPM cell. 

First, you need to decide what BPPM cell class you want to map to.  You can map to base class EVENT or an existing subclass of EVENT such as PATROL_EV.  Or, you can define your own class.  I personally prefer to define a specific class for each 3rd-party monitoring software regardless if I need to add extra slots to it or not.  For example, I have defined SCOM_EV for events from Microsoft SCOM and NAGIOS_EV for events from Open Source tool Nagios.

Second, you need to decide what slots in BPPM cell need to be mapped.  Each event in BPPM cell has more than 75 slots.  Obviously you don't need map all 75 slots.  If you will send emails for some events, the slots referenced in your email will need to be mapped.  If you will create incident tickets from some events, the slots that map to your ticketing system will need to be mapped from your 3rd-party monitoring software.  If you have a NOC team that watches BPPM GUI to help create and escalate tickets, the slots your NOC team can see will need to be mapped. Commonly mapped BPPM cell slots include mc_host, mc_object_class, mc_object, mc_tool_class, mc_tool_key, mc_parameter, mc_parameter_value, severity, mc_priority, and msg.

Third, you need to decide how to map. For most of BPPM cell slots, you can map directly.  For example, you can use mc_host=${HOSTNAME} to map HOSTNAME field from your 3rd-party monitoring software directly to BPPM cell slot mc_host.  However for some slots in BPPM cell, especially the ones in enumerable type, you may need to perform value conversion unless BPPM cell and your 3rd-party monitoring software use exactly the same spelling for all their values. For example, if your 3rd-party monitoring software uses high, medium, and low as the value of event priority, but BPPM cell uses PRIORITY_1 to PRIORITY_5 as the values of events priority, you would need to convert the values in your 3rd-party monitoring software to the values in BPPM cell.

You can perform conversion in your OS script. Or you can map the conversion-required fields to other string-typed BPPM slots 'as is'.  Then you can create a MRL rule in BPPM cell to perform the conversion.  In the above example, you can create a new BPPM cell slot called priority_string to map priority field from your 3rd-party monitoring software.  Then in your MRL rule, you can convert value in priority_string slot to mc_priority slot.  By performing conversion in BPPM cell, you can take advantage of dynamic tables to make value conversion easy to code and easy to maintain.

Event Integration From 3rd-party Monitoring Tools - Part 4: Msend API execution

After installing msend API package and successfully testing the communication between your 3rd-party monitoring software server and your BPPM/BEM cell, you are ready to call msend from your 3rd-party monitoring software.

In your 3rd-party monitoring software, first locate where you can configure an OS script when an alert is raised.  Usually you will need to specify a condition (such as a specific severity or a specific component) and the name of an OS script.

Although you can directly call msend by using 'msend' as your OS script name, I recommend that you create a separate OS script (e.g. call_msend).  Your 3rd-party monitoring software will call this OS script when an alert is raised.  Then this OS script will call msend.  This way you can set the environment to execute msend inside your OS script without having to merge msend environment variables into your 3rd-party monitoring software environment - which would often require you to restart your 3rd-party monitoring software.  In addition, you can unit test your OS script separately without having to wait for an alert to happen.

Second, decide which system variables (also called micros in some 3rd-party monitoring software) you would like to send to BPPM/BEM cell. These system variables contain the values of hostname, component name, parameter name, parameter value, etc. from the raised alert. The values of these system variables will be used when BPPM/BEM cell sends a notification email or creates an incident ticket.  You will configure these system variables as arguments to your OS script.

Third, construct your OS script. In the script, set up environment to execute msend, use buffer mode (-j) for guaranteed delivery, and decide to send events from command line or a text file. Examples of sending events in buffer mode, sending events from command line, and sending events from a text file were listed in the previous post "Event Integration From 3rd-party Monitoring Tools - Part 3: Msend API Configuration".

Sending events from command line is a little faster than sending events from a text file because no file I/O is involved, but sending events from a text file is easier for trouble shooting since you can keep each text file as a sender's log.  If you do choose to send events from a text file, make sure that you always use a unique file name for each event.  You can delete the text file immediately after sending the event or manually delete them later when you no longer need them for trouble shooting purpose.

In the next post, we will discuss mapping and conversion when using msend API.

Event Integration From 3rd-party Monitoring Tools - Part 3: Msend API configuration

Before you install msend API package, on the server where your 3rd-party monitoring software runs, decide your installation directory <Install_Dir> and define the following environment variable:

$MCELL_HOME=<Install_Dir>/BMC/Server (for UNIX) or

%MCELL_HOME%=<Install_Dir>\BMC\Server (for Windows)

Then add $MCELL_HOME/bin or %MCELL_HOME%\bin to the environment variable PATH.

Now you can untar or unzip your msend API package under %MCELL_HOME% directory.  After all files are installed, open %MCELL_HOME%\etc\mcell.dir and verify all cells are defined correctly.  Then create a directory called "buffer" under %MCELL_HOME%.

For UNIX, verify that $MCELL_HOME/bin/msend has its execution bit turned on.  Run 'chmod +x msend' if needed. In addition, in order to run msend API in buffer mode, you need to create a file called /etc/itm/.reg and make /etc/itm/.reg writable to the account executing msend API.  You need root privilege to create /etc/itm/.reg.

Before calling msend API from your 3rd-party monitoring software, I highly recommend testing the communication by using the following simple test cases:

1. Send an event from command line: 

    msend -n <cell_name> -a EVENT -m "Test from command line"

2. Send an event from command line in buffer mode:

    msend -n <cell_name> -j <Install_Dir>\BMC\server\buffer -a EVENT -m "Test from command line with buffer"

3. Send an event from a text file:

    msend -n <cell_name> myevent1.txt

where myevent1.txt should contain these lines:

EVENT;

       severity=CRITICAL;

       msg="Test from text file";

END

4. Send an event from a text file in buffer mode:

    msend -n <cell_name> -j <Install_Dir>\BMC\server\buffer myevent2.txt

where myevent2.txt should contain these lines:

EVENT;

       severity=CRITICAL;

       msg="Test from text file with buffer";

END

   

In each of the above four cases, verify that an event is present in BPPM/BEM GUI and no error message is displayed from msend command. If you are running msend in buffer mode, also verify that 'msend' process is still running up to 10 minutes after you call msend command.

We will cover msend API execution in the next post.