Total Cost of Ownership of BPPM - Part 13: Best practice - When not to use out-of-box default values

Happy New Year to everyone! 

What an exciting year 2014 was with both BMC BPPM 9.5 and BMC TrueSight 9.6 going strong!  I want to take this opportunity to thank you for your continuous trust and support.  May 2015 bring you more success and happiness. 

To continue our discussion on best practices regarding to the total cost of ownership of BPPM, we are going to talk about when you should not use out-of-box default values in your BPPM implementation.  Since there will be almost endless situations when using the default values is not a good idea, let's take a look at three different examples.

The first example is the installation directory path. The default installation directory path for BPPM installation is C:\Program Files\BMC Software\.  The spaces used here can potentially make your customization scripts fail.  The long path name can potentially truncate your file name in those deep-down sub-directories.  In addition, you should use a disk drive other than C to lower BPPM's performance impact on Operating System.  Assuming you have a E drive available, a much better installation directory path for BPPM installation would be: E:\BMC\.

The second example is the remote cell names. The default remote cell name supplied by BPPM installer is the hostname where the cell is being installed.  If you are setting your remote cells in an application cluster, you need to use the same cell name for its backup cell running on another host.  For example, when you are installing remote cells in an application cluster on server A and B, if you take the default cell name, you would name the cell as 'A' for both cells running on server A and server B.  This would cause a lot of confusion when configuring the event sources sending events to this cell, not to mention most hostnames are very hard to remember.  A better way to name your remote cells is to name it based on environment, function, and/or H/A type.  For example, I would name my remote cell receiving Entuity events in an application cluster running on production environment as PRD_ENTUITY_HA.

The third example is the event database size for a remote cell.  By default, event database size in mcell.conf is set at 360,000 (EventDBSize=360000).  Each PATROL event is about 4K.  For 360K events, the maximum event DB size would be 360K * 4K = 1.44GB. If this size is more than half of the RAM size on the server, you would most likely see a long running time for cell's state builder and your cell would be nonresponsive while its state builder is running.  Ensure your EventDBSize is less than 50% of RAM size.  If you are running multiple cells or sharing the server with BPPM integration service, you need to be even more conservative.

Total Cost of Ownership of BPPM - Part 12: Best practice - Fail to plan means plan to fail

I recently took a client project of BMC TrueSight/BPPM 9.6 implementation.  The client insisted to do the installation themselves and had me involved after the installation was completed.  When I was finally involved with the project after they spent weeks to finish the installation, I found out that many things were installed incorrectly: The BPPM server was under capacity, BPPM and Entuity were sharing the same server, the 2 BPPM Integration Service servers and 2 BPPM Cell servers were each configured as application cluster in an Windows OS cluster.  On top of that, there was no written architecture plan.

Needless to say, the installation needed to be re-done.  Had they had an architecture drawing or written plan before starting installation, and let me (or someone with BPPM architecture experience) review and confirm it, they could have saved themselves a lot of time in installation.

So what need to be planned before you start installation?

1) Server capacity: BPPM server is a heavy duty server.  It includes web server, app server, analytic engine, BPPM cell, and optional Sybase database.  All these components are memory "hungry".  Unless you are installing for a proof-of-concept environment, go for 32GB RAM.  If you are implementing both BPPM and Entuity, install Entuity on a separate server.  With a MySQL database on the same server, Entuity needs to have 32GB RAM too.  Virtual machines work OK though physical machines would give you more power.  BPPM Integration Service server is much less demanding and would work OK with 8GB RAM.  Each Integration Service (pair) can connect to 900 PATROL agents.  BPPM cell server needs 4GB RAM for one cell or 8GB RAM for 2-4 cells.

2) High availability: There are three different options for high availability - application cluster, VMWare cluster, and OS cluster (such as Microsoft Windows Cluster).  Application cluster can fail over at individual process level when a process fails.  OS cluster can fail over at host level when a process fails.  VMWare cluster can fail over at host level when a host fails.  For BPPM Integration Service servers and BPPM cell servers, configure them as application clusters if you have enough dedicated hardware.  Do not mix an application cluster with OS cluster.  For BPPM servers, since they don't support application cluster, use OS cluster if you have enough dedicated hardware.  If you don't have enough dedicated hardware and run everything on virtual machines, you can configure VMWare cluster on all servers. 

3) Integration: BPPM server can be integrated with many other BMC products such as BMC Sever Automation, Atrium CMDB, and Single Sign On.  If you plan to configure any of these integrations during BPPM server installation, you need to have detailed integration information ready.  However you do have an option to configure your integrations later.

4) Name convention: In additional to installation directory path, you need to have a systematic way to determine the name for each BPPM cell, BPPM integration service, and BPPM integration service cluster.  More on name convention planning was discussed in this earlier post: BPPM 9.5 Quick Start - Part 3: Name convention

Total Cost of Ownership of BPPM - Part 11: Best practice - Document implementation with check points

You were excited to start implementing a new version of BPPM.  You spent days or weeks reading through BMC documents, communicating with your system administrators to get the servers ready, communicating with your DBAs to get your database ready, navigating through complicated installation processes with various options regarding to high availability and integration, and you may have had to open cases with BMC Support to finally finish installing different BPPM components on different systems.

Then when you started all BPPM components and log into BPPM GUI, you realized it didn't work - BPPM components didn't communicate with each other, data or events didn't show up in BPPM GUI, and several errors were seen in log files.  Which part went wrong?  You ended up opening another case with BMC Support.

Sounds familiar?  What is worse is that, after you spend another week or so working with BMC Support to finally figure out what went wrong and make the correction, the similar headache may happen again when you start BPPM implementation in another environment.

This situation can be improved or even avoided by adding check points in your implementation process.  Check points are verification steps to perform after completing installation or configuration of partial or one BPPM component.  Check points give you confirmation of "so far so good" before moving into the next step in implementation.

For example, before you install BPPM server, a check point should be running some SQL statements to verify correct Oracle user accounts and tablespaces are created. When finishing installing a H/A pair of BPPM Integration Services and one PATROL agent, another check point should be verifying that PATROL agent can fail over to the secondary BPPM Integration Service when the primary one goes down.

Check points help you notice the problems in your installation or configuration so you can correct them as they occur.  If a check point fails, it is a whole lot easier to find out what went wrong and correct it right there than realizing things don't work at the end of implementation.

In addition, it is highly recommended that you write your own version of installation and configuration documents with all check points.  The installation and configuration documents from BMC are generic version.  It will take you much longer to read them and apply them to your own environment.

When implementing a new version of BPPM for the first time, write down all the steps you performed and information you entered to create your own version of installation and configuration documents.  I personally like to save all screen shots in the documents because one picture is worth a thousand words.  And of course, always include detailed steps for check points.

This may seem to be time consuming first especially when you are under the pressure to have the installation "done" before the deadline.  But having your own version of installation and configuration documents will drastically speed up future deployment and trouble-shooting process.  My experience from working on BMC products for the last 20 years has proven to me that it is totally worth the effort.

Total Cost of Ownership of BPPM - Part 10: Best practice - GUI vs Batch

BPPM has provided many different graphical user interfaces to help you configure different BPPM components.  For example, you can use PATROL Central Console or PATROL Classical Console to configure a PATROL KM.  The GUI provides a way to validate many things you typed including directory names, Oracle instance names, etc.

GUI is great to configure your first server, your second server, perhaps not even bad on your fifth server.  How about if you have 200 Oracle servers to configure? Don't you wish you have a less tedious and faster way to accomplish your work?

Yes when it comes to mass deployment, batch would be a whole lot faster and easier to accomplish your work and keep you from going nuts with repetitive work. Here is how you can move from GUI to batch for your PATROL KM configuration.

For PATROL KM configuration, on the first server, you run 'pconfig +get > C:\temp\pconfig1.txt' command to retrieve all pconfig values into a text file. Then you simply configure the KM on your first server using GUI.  After that run 'pconfig +get > C:\temp\pconfig2.txt' command to retrieve all pconfig values into another text file.  Do a diff on these two text files, you will have all pconfig variables that belong to this KM.  Look at each pconfig variable to find out what needs to be changed for a different server.  If you are not sure, repeat the same process on your second server to confirm what values need to be changed.  Simply make the changes either manually or using a script.  You will end up having different pconfig sets to be deployed to different servers using either 'pconfig' command or PATROL Configuration Manager (PCM).

Where else can you use batch deployment?  You may need to use batch deployment with BPPM cell since you may have total 10-100 cells in your dev, QA, and production environment combined.  Many of your BPPM cells have the same policies and data tables that are configured through GUI.  In this case, you can simply export the policies and data tables from an existing cell, make any necessary modifications, and import them into the new cell.  Using batch deployment on BPPM cell configuration will not only cut down your deployment time, it will also eliminate discrepancies due to human errors.

In general, any GUI configuration in your environment is worth finding a way for batch deployment if the work needs to be repeated more than 3 times.

Total Cost of Ownership of BPPM - Part 9: Best practice - Store only one copy of the same file

In the previous post, we discussed file repository where you store all your customization files. It simplifies deployment because you always know where the most up-to-date copy of each file is located.  However if you keep more than one copy of the same file in your file repository, these multiple copies can become out of sync and then you have to figure out which one is the most up-to-date copy when you deploy.

How can you end up having multiple copies of the same file in your file repository? Different server roles and different environments can require different file content with the same file name.  For example, .load file would be different for your PATROL cell and for your correlation cell.  IBRSD.conf file would be different for your development environment and for your production environment. 

Does this mean that you have to have separate file repositories for different cells and different environments?  No.  Since most of the files are still the same in both your development environment and your production environment, having separate file repositories can make them out of sync quickly.

Because only small percentage of customization files are different in different environments, I simply use a renaming method to store and deploy them.  For example, since IBRSD.conf is different in development environment and in production environment while most of other IBRSD files are the same for both environments, I store both copies of IBRSD.conf in my file repository as IBRSD.conf.dev and IBRSD.conf.prd in the same directory.  When I deploy IBRSD to a production server, both IBRSD.conf.dev and IBRSD.conf.prd will be copied to the destination server.  Then I will run a post deployment script to rename any file with a .prd extension to its original file name without .prd extension.

By using this renaming method, I can handle a small number of file variations while avoiding to store multiple copies of the same file in my file repository.  This method also keeps the size of file repository small.

Having a small file repository means that you can back it up on a thumb drive as the files are mostly ASCII.  In a catastrophic situation, you can rebuild your entire system quickly at another location with exactly the same customization you had at your original location.

Total Cost of Ownership of BPPM - Part 8: Best practice - File repository

During BPPM implementation, you will need to create some of your own files or modify out-of-box BPPM files.  These customization files include configuration files, cmd files, shell scripts, Perl scripts, KM files, PSL files, BAROC files, MRL files, etc.

One of the common practices I have seen at many organizations is to simply create new files and leave them on the server.  When the same files are needed for the second server, they are copied from the first server.  When the number of files becomes large, individual file copy is replaced by directory copy.

This practice certainly works in some extent, but there are several issues. For example,

1) If you make modification to some of the files while working on the third server, you will need to remember to repeat the same modification on the first and second servers.  Otherwise, your files on those servers will be out of sync.  A few months later, you won't remember which server contains the most up-to-date copy of your customization files.

2) When you start copying the entire directory instead of individual files, you could copy all the junk files generated during testing to the next server.  A few months later, you will have a hard time to remember what those junk files are for.

3) Upgrade will be a nightmare.  You will need to figure out which files you need to copy into the next version, which files you need to merge, and how many different variations of configuration/data files you need to extract out of the current version.

By using a single file repository, you can minimize even eliminate the above issues.

Pick up a shared drive as your file repository at a location where you can easily copy files to and from other servers.  Keep the directory structure as identical as possible to the destination servers (such as server\etc\cellname\kb\rules\).  Save only the files you created or modified in the file repository.  Never copy the original BPPM out-of-box files there.

As soon as you finish testing your customization files on a server, always save a copy of most up-to-date files into your file repository.  When you need to deploy files to a new server, always copy files from the file repository, not from another server.  If you are not sure if a server has the most up-to-date version of your customization, simply deploy all your files again from the file repository.  When it is time to upgrade, the files in your repository are the only files you you need to copy/merge into the next version.

Total Cost of Ownership of BPPM - Part 7: Best practice - Name convention

Many things you can do in BPPM implementation to lower the total cost of ownership of BPPM.  Some of them are big such as architecture design.  And some of them seem to be small such as name convention.  However small they seem to be, they help minimize 'Winchester House' syndrome and lower the total cost of ownership of BPPM.

I would like to start best practice discussion with name convention.  BPPM doesn't post any requirement on how you name your BPPM components including files, cells, integration services, clusters, CMA tags, etc.  If there is no name convention in place, each person would end up naming BPPM components any way he/she likes.  Sooner or later, your BPPM implementation would become a Winchester House. 

Having a name convention in place at the beginning of BPPM implementation is a small effort but it pays many times back after the implementation is completed.  The name convention needs to be enforced at all time during the implementation and after the implementation.  Trying to rename things afterwards is a painful and error-prone process.

Pick up a 2 or 3-letter prefix for your organization.  It will help distinguish your custom files from BPPM out-of-box files.  For example, I used prefix C1 for CapitalOne and prefix CTI for CitiMortgage. Use this prefix for your custom PATROL KM files, PSL files, cell knowledge base MRL files, BAROC files, shell scripts, perl scripts, batch files, JAVA scripts, etc.

For custom PATROL KM, also pick up a short name for your KM.  I also prefer to use all capitals for KM short names.  Include this short name in all file names related to the KM.  This will make your KM easy to package and deploy without missing any file.  If the KM requires specific pconfig file, I would use the same name convention for the pconfig file.  For example, here are the file names I used for a custom CACHE KM: XYZ_CACHE_main.km, XYZ_CACHE_db.km, XYZ_CACHE_db_collector.psl.  XYZ_CACHE.cfg.  XYZ here is the organization prefix. 

For cell names, never ever reuse the same name in the entire organization.  Since cell names are prompted during BPPM installation, you need to have a discussion and decide on their names before you start installation.  The information needs to be included in cell names include: 1) Environment: dev, QA, or production. 2) Event source: PATROL, SCOM, external, correlation, server, etc. 3) Type: H/A or standalone.  For example, I named the cell located on the first clustered PATROL integration service in dev environment as Dev_PATROL1_HA. 

CMA tags also need to be determined in advance.  It can usually be divided into two different ways: by infrastructure (OS, DB, application) and by user environment. For example, you may have CMA tags like Windows_Base, Oracle_Canada, Peoplesoft_HR, etc.  The less overlap between tags, the easier to use and maintain them.  Be careful when making changes to existing CMA tags after implementation is completed because all PATROL agents with the same tag names will have their pconfig updated automatically.