Event Integration From 3rd-party Monitoring Tools - Part 6: SNMP trap adapter installation

SNMP trap adapter is another way to integrate events from your 3rd-party monitoring software into a BPPM/BEM cell. About 90% of monitoring software out there can send SNMP traps when alerts are raised. SNMP trap adapter is a perl-based adapter provided by BMC to translate SNMP traps to events in BAROC format, a format that BPPM/BEM cell can understand.  By default, SNMP trap adapter listens on port 162.

SNMP trap adapter is part of BMC ProactiveNet Event Adapters. It is often installed on the same server where BPPM/BEM cell is installed.  However, it is not required to have SNMP trap adapter and the BPPM/BEM cell it connects to co-locate on the same server.

Although BMC talks about SNMP trap adapter all the time and barely mention msend API, I personally prefer msend API whenever possible and use SNMP trap adapter only if the 3rd-party monitoring software is not capable of executing an OS script when an alert is raised. When buffer mode '-j' is used, events sent using msend API will be guaranteed to arrive at the BPPM cell.  If the cell is down or unreachable when your 3rd-party monitoring software is calling msend API, msend will buffer the unsent events and keep trying until the cell is up.  If SNMP trap adapter is down or unreachable when your 3rd-party monitoring software is sending SNMP traps, all SNMP traps will be lost.

Before starting SNMP trap adapter, edit mcxa.conf.  Locate [Snmp] section.  If you use SNMP version 3, locate [SnmpV3] section. Remove the word 'DISABLE'.  Add the following lines to maximize SNMP trap throughput.

PollInterval     = 1

ReadsPerEngine   = 200

SnmpRcvbuf       = 262144

If ServerName is not set to your cell name under [default] section, you also need to set ServerName to your cell name under [Snmp] section.

Now you can start your SNMP trap adapter.  Check the adapter status again after a couple of minutes and verify it stays up.  If you need to go back and change mcxa.conf and restart SNMP trap adapter, wait for a couple of minutes between stop and start the adapter to give it enough time to release port 162.

We will discuss configuration, execution, mapping, and conversion in the next few posts.

Event Integration From 3rd-party Monitoring Tools - Part 5: Msend API mapping and conversion

In your 3rd-party monitoring software, each event (also called alert) contains many different fields (also called attributes).  You need to map those field names to BPPM cell slot names in order to pass those field values into BPPM cell. 

First, you need to decide what BPPM cell class you want to map to.  You can map to base class EVENT or an existing subclass of EVENT such as PATROL_EV.  Or, you can define your own class.  I personally prefer to define a specific class for each 3rd-party monitoring software regardless if I need to add extra slots to it or not.  For example, I have defined SCOM_EV for events from Microsoft SCOM and NAGIOS_EV for events from Open Source tool Nagios.

Second, you need to decide what slots in BPPM cell need to be mapped.  Each event in BPPM cell has more than 75 slots.  Obviously you don't need map all 75 slots.  If you will send emails for some events, the slots referenced in your email will need to be mapped.  If you will create incident tickets from some events, the slots that map to your ticketing system will need to be mapped from your 3rd-party monitoring software.  If you have a NOC team that watches BPPM GUI to help create and escalate tickets, the slots your NOC team can see will need to be mapped. Commonly mapped BPPM cell slots include mc_host, mc_object_class, mc_object, mc_tool_class, mc_tool_key, mc_parameter, mc_parameter_value, severity, mc_priority, and msg.

Third, you need to decide how to map. For most of BPPM cell slots, you can map directly.  For example, you can use mc_host=${HOSTNAME} to map HOSTNAME field from your 3rd-party monitoring software directly to BPPM cell slot mc_host.  However for some slots in BPPM cell, especially the ones in enumerable type, you may need to perform value conversion unless BPPM cell and your 3rd-party monitoring software use exactly the same spelling for all their values. For example, if your 3rd-party monitoring software uses high, medium, and low as the value of event priority, but BPPM cell uses PRIORITY_1 to PRIORITY_5 as the values of events priority, you would need to convert the values in your 3rd-party monitoring software to the values in BPPM cell.

You can perform conversion in your OS script. Or you can map the conversion-required fields to other string-typed BPPM slots 'as is'.  Then you can create a MRL rule in BPPM cell to perform the conversion.  In the above example, you can create a new BPPM cell slot called priority_string to map priority field from your 3rd-party monitoring software.  Then in your MRL rule, you can convert value in priority_string slot to mc_priority slot.  By performing conversion in BPPM cell, you can take advantage of dynamic tables to make value conversion easy to code and easy to maintain.

Event Integration From 3rd-party Monitoring Tools - Part 4: Msend API execution

After installing msend API package and successfully testing the communication between your 3rd-party monitoring software server and your BPPM/BEM cell, you are ready to call msend from your 3rd-party monitoring software.

In your 3rd-party monitoring software, first locate where you can configure an OS script when an alert is raised.  Usually you will need to specify a condition (such as a specific severity or a specific component) and the name of an OS script.

Although you can directly call msend by using 'msend' as your OS script name, I recommend that you create a separate OS script (e.g. call_msend).  Your 3rd-party monitoring software will call this OS script when an alert is raised.  Then this OS script will call msend.  This way you can set the environment to execute msend inside your OS script without having to merge msend environment variables into your 3rd-party monitoring software environment - which would often require you to restart your 3rd-party monitoring software.  In addition, you can unit test your OS script separately without having to wait for an alert to happen.

Second, decide which system variables (also called micros in some 3rd-party monitoring software) you would like to send to BPPM/BEM cell. These system variables contain the values of hostname, component name, parameter name, parameter value, etc. from the raised alert. The values of these system variables will be used when BPPM/BEM cell sends a notification email or creates an incident ticket.  You will configure these system variables as arguments to your OS script.

Third, construct your OS script. In the script, set up environment to execute msend, use buffer mode (-j) for guaranteed delivery, and decide to send events from command line or a text file. Examples of sending events in buffer mode, sending events from command line, and sending events from a text file were listed in the previous post "Event Integration From 3rd-party Monitoring Tools - Part 3: Msend API Configuration".

Sending events from command line is a little faster than sending events from a text file because no file I/O is involved, but sending events from a text file is easier for trouble shooting since you can keep each text file as a sender's log.  If you do choose to send events from a text file, make sure that you always use a unique file name for each event.  You can delete the text file immediately after sending the event or manually delete them later when you no longer need them for trouble shooting purpose.

In the next post, we will discuss mapping and conversion when using msend API.

Event Integration From 3rd-party Monitoring Tools - Part 3: Msend API configuration

Before you install msend API package, on the server where your 3rd-party monitoring software runs, decide your installation directory <Install_Dir> and define the following environment variable:

$MCELL_HOME=<Install_Dir>/BMC/Server (for UNIX) or

%MCELL_HOME%=<Install_Dir>\BMC\Server (for Windows)

Then add $MCELL_HOME/bin or %MCELL_HOME%\bin to the environment variable PATH.

Now you can untar or unzip your msend API package under %MCELL_HOME% directory.  After all files are installed, open %MCELL_HOME%\etc\mcell.dir and verify all cells are defined correctly.  Then create a directory called "buffer" under %MCELL_HOME%.

For UNIX, verify that $MCELL_HOME/bin/msend has its execution bit turned on.  Run 'chmod +x msend' if needed. In addition, in order to run msend API in buffer mode, you need to create a file called /etc/itm/.reg and make /etc/itm/.reg writable to the account executing msend API.  You need root privilege to create /etc/itm/.reg.

Before calling msend API from your 3rd-party monitoring software, I highly recommend testing the communication by using the following simple test cases:

1. Send an event from command line: 

    msend -n <cell_name> -a EVENT -m "Test from command line"

2. Send an event from command line in buffer mode:

    msend -n <cell_name> -j <Install_Dir>\BMC\server\buffer -a EVENT -m "Test from command line with buffer"

3. Send an event from a text file:

    msend -n <cell_name> myevent1.txt

where myevent1.txt should contain these lines:

EVENT;

       severity=CRITICAL;

       msg="Test from text file";

END

4. Send an event from a text file in buffer mode:

    msend -n <cell_name> -j <Install_Dir>\BMC\server\buffer myevent2.txt

where myevent2.txt should contain these lines:

EVENT;

       severity=CRITICAL;

       msg="Test from text file with buffer";

END

   

In each of the above four cases, verify that an event is present in BPPM/BEM GUI and no error message is displayed from msend command. If you are running msend in buffer mode, also verify that 'msend' process is still running up to 10 minutes after you call msend command.

We will cover msend API execution in the next post.