I recently took a client project of BMC TrueSight/BPPM 9.6 implementation. The client insisted to do the installation themselves and had me involved after the installation was completed. When I was finally involved with the project after they spent weeks to finish the installation, I found out that many things were installed incorrectly: The BPPM server was under capacity, BPPM and Entuity were sharing the same server, the 2 BPPM Integration Service servers and 2 BPPM Cell servers were each configured as application cluster in an Windows OS cluster. On top of that, there was no written architecture plan.
Needless to say, the installation needed to be re-done. Had they had an architecture drawing or written plan before starting installation, and let me (or someone with BPPM architecture experience) review and confirm it, they could have saved themselves a lot of time in installation.
So what need to be planned before you start installation?
1) Server capacity: BPPM server is a heavy duty server. It includes web server, app server, analytic engine, BPPM cell, and optional Sybase database. All these components are memory "hungry". Unless you are installing for a proof-of-concept environment, go for 32GB RAM. If you are implementing both BPPM and Entuity, install Entuity on a separate server. With a MySQL database on the same server, Entuity needs to have 32GB RAM too. Virtual machines work OK though physical machines would give you more power. BPPM Integration Service server is much less demanding and would work OK with 8GB RAM. Each Integration Service (pair) can connect to 900 PATROL agents. BPPM cell server needs 4GB RAM for one cell or 8GB RAM for 2-4 cells.
2) High availability: There are three different options for high availability - application cluster, VMWare cluster, and OS cluster (such as Microsoft Windows Cluster). Application cluster can fail over at individual process level when a process fails. OS cluster can fail over at host level when a process fails. VMWare cluster can fail over at host level when a host fails. For BPPM Integration Service servers and BPPM cell servers, configure them as application clusters if you have enough dedicated hardware. Do not mix an application cluster with OS cluster. For BPPM servers, since they don't support application cluster, use OS cluster if you have enough dedicated hardware. If you don't have enough dedicated hardware and run everything on virtual machines, you can configure VMWare cluster on all servers.
3) Integration: BPPM server can be integrated with many other BMC products such as BMC Sever Automation, Atrium CMDB, and Single Sign On. If you plan to configure any of these integrations during BPPM server installation, you need to have detailed integration information ready. However you do have an option to configure your integrations later.
4) Name convention: In additional to installation directory path, you need to have a systematic way to determine the name for each BPPM cell, BPPM integration service, and BPPM integration service cluster. More on name convention planning was discussed in this earlier post: BPPM 9.5 Quick Start - Part 3: Name convention
Monday, December 29, 2014
Monday, December 22, 2014
Total Cost of Ownership of BPPM - Part 11: Best practice - Document implementation with check points
You were excited to start implementing a new version of BPPM. You spent days or weeks reading through BMC documents, communicating with your system administrators to get the servers ready, communicating with your DBAs to get your database ready, navigating through complicated installation processes with various options regarding to high availability and integration, and you may have had to open cases with BMC Support to finally finish installing different BPPM components on different systems.
Then when you started all BPPM components and log into BPPM GUI, you realized it didn't work - BPPM components didn't communicate with each other, data or events didn't show up in BPPM GUI, and several errors were seen in log files. Which part went wrong? You ended up opening another case with BMC Support.
Sounds familiar? What is worse is that, after you spend another week or so working with BMC Support to finally figure out what went wrong and make the correction, the similar headache may happen again when you start BPPM implementation in another environment.
This situation can be improved or even avoided by adding check points in your implementation process. Check points are verification steps to perform after completing installation or configuration of partial or one BPPM component. Check points give you confirmation of "so far so good" before moving into the next step in implementation.
For example, before you install BPPM server, a check point should be running some SQL statements to verify correct Oracle user accounts and tablespaces are created. When finishing installing a H/A pair of BPPM Integration Services and one PATROL agent, another check point should be verifying that PATROL agent can fail over to the secondary BPPM Integration Service when the primary one goes down.
Check points help you notice the problems in your installation or configuration so you can correct them as they occur. If a check point fails, it is a whole lot easier to find out what went wrong and correct it right there than realizing things don't work at the end of implementation.
In addition, it is highly recommended that you write your own version of installation and configuration documents with all check points. The installation and configuration documents from BMC are generic version. It will take you much longer to read them and apply them to your own environment.
When implementing a new version of BPPM for the first time, write down all the steps you performed and information you entered to create your own version of installation and configuration documents. I personally like to save all screen shots in the documents because one picture is worth a thousand words. And of course, always include detailed steps for check points.
This may seem to be time consuming first especially when you are under the pressure to have the installation "done" before the deadline. But having your own version of installation and configuration documents will drastically speed up future deployment and trouble-shooting process. My experience from working on BMC products for the last 20 years has proven to me that it is totally worth the effort.
Then when you started all BPPM components and log into BPPM GUI, you realized it didn't work - BPPM components didn't communicate with each other, data or events didn't show up in BPPM GUI, and several errors were seen in log files. Which part went wrong? You ended up opening another case with BMC Support.
Sounds familiar? What is worse is that, after you spend another week or so working with BMC Support to finally figure out what went wrong and make the correction, the similar headache may happen again when you start BPPM implementation in another environment.
This situation can be improved or even avoided by adding check points in your implementation process. Check points are verification steps to perform after completing installation or configuration of partial or one BPPM component. Check points give you confirmation of "so far so good" before moving into the next step in implementation.
For example, before you install BPPM server, a check point should be running some SQL statements to verify correct Oracle user accounts and tablespaces are created. When finishing installing a H/A pair of BPPM Integration Services and one PATROL agent, another check point should be verifying that PATROL agent can fail over to the secondary BPPM Integration Service when the primary one goes down.
Check points help you notice the problems in your installation or configuration so you can correct them as they occur. If a check point fails, it is a whole lot easier to find out what went wrong and correct it right there than realizing things don't work at the end of implementation.
In addition, it is highly recommended that you write your own version of installation and configuration documents with all check points. The installation and configuration documents from BMC are generic version. It will take you much longer to read them and apply them to your own environment.
When implementing a new version of BPPM for the first time, write down all the steps you performed and information you entered to create your own version of installation and configuration documents. I personally like to save all screen shots in the documents because one picture is worth a thousand words. And of course, always include detailed steps for check points.
This may seem to be time consuming first especially when you are under the pressure to have the installation "done" before the deadline. But having your own version of installation and configuration documents will drastically speed up future deployment and trouble-shooting process. My experience from working on BMC products for the last 20 years has proven to me that it is totally worth the effort.
Monday, December 15, 2014
Total Cost of Ownership of BPPM - Part 10: Best practice - GUI vs Batch
BPPM has provided many different graphical user interfaces to help you configure different BPPM components. For example, you can use PATROL Central Console or PATROL Classical Console to configure a PATROL KM. The GUI provides a way to validate many things you typed including directory names, Oracle instance names, etc.
GUI is great to configure your first server, your second server, perhaps not even bad on your fifth server. How about if you have 200 Oracle servers to configure? Don't you wish you have a less tedious and faster way to accomplish your work?
Yes when it comes to mass deployment, batch would be a whole lot faster and easier to accomplish your work and keep you from going nuts with repetitive work. Here is how you can move from GUI to batch for your PATROL KM configuration.
For PATROL KM configuration, on the first server, you run 'pconfig +get > C:\temp\pconfig1.txt' command to retrieve all pconfig values into a text file. Then you simply configure the KM on your first server using GUI. After that run 'pconfig +get > C:\temp\pconfig2.txt' command to retrieve all pconfig values into another text file. Do a diff on these two text files, you will have all pconfig variables that belong to this KM. Look at each pconfig variable to find out what needs to be changed for a different server. If you are not sure, repeat the same process on your second server to confirm what values need to be changed. Simply make the changes either manually or using a script. You will end up having different pconfig sets to be deployed to different servers using either 'pconfig' command or PATROL Configuration Manager (PCM).
Where else can you use batch deployment? You may need to use batch deployment with BPPM cell since you may have total 10-100 cells in your dev, QA, and production environment combined. Many of your BPPM cells have the same policies and data tables that are configured through GUI. In this case, you can simply export the policies and data tables from an existing cell, make any necessary modifications, and import them into the new cell. Using batch deployment on BPPM cell configuration will not only cut down your deployment time, it will also eliminate discrepancies due to human errors.
In general, any GUI configuration in your environment is worth finding a way for batch deployment if the work needs to be repeated more than 3 times.
GUI is great to configure your first server, your second server, perhaps not even bad on your fifth server. How about if you have 200 Oracle servers to configure? Don't you wish you have a less tedious and faster way to accomplish your work?
Yes when it comes to mass deployment, batch would be a whole lot faster and easier to accomplish your work and keep you from going nuts with repetitive work. Here is how you can move from GUI to batch for your PATROL KM configuration.
For PATROL KM configuration, on the first server, you run 'pconfig +get > C:\temp\pconfig1.txt' command to retrieve all pconfig values into a text file. Then you simply configure the KM on your first server using GUI. After that run 'pconfig +get > C:\temp\pconfig2.txt' command to retrieve all pconfig values into another text file. Do a diff on these two text files, you will have all pconfig variables that belong to this KM. Look at each pconfig variable to find out what needs to be changed for a different server. If you are not sure, repeat the same process on your second server to confirm what values need to be changed. Simply make the changes either manually or using a script. You will end up having different pconfig sets to be deployed to different servers using either 'pconfig' command or PATROL Configuration Manager (PCM).
Where else can you use batch deployment? You may need to use batch deployment with BPPM cell since you may have total 10-100 cells in your dev, QA, and production environment combined. Many of your BPPM cells have the same policies and data tables that are configured through GUI. In this case, you can simply export the policies and data tables from an existing cell, make any necessary modifications, and import them into the new cell. Using batch deployment on BPPM cell configuration will not only cut down your deployment time, it will also eliminate discrepancies due to human errors.
In general, any GUI configuration in your environment is worth finding a way for batch deployment if the work needs to be repeated more than 3 times.
Monday, December 8, 2014
Total Cost of Ownership of BPPM - Part 9: Best practice - Store only one copy of the same file
In the previous post, we discussed file repository where you store all your customization files. It simplifies deployment because you always know where the most up-to-date copy of each file is located. However if you keep more than one copy of the same file in your file repository, these multiple copies can become out of sync and then you have to figure out which one is the most up-to-date copy when you deploy.
How can you end up having multiple copies of the same file in your file repository? Different server roles and different environments can require different file content with the same file name. For example, .load file would be different for your PATROL cell and for your correlation cell. IBRSD.conf file would be different for your development environment and for your production environment.
Does this mean that you have to have separate file repositories for different cells and different environments? No. Since most of the files are still the same in both your development environment and your production environment, having separate file repositories can make them out of sync quickly.
Because only small percentage of customization files are different in different environments, I simply use a renaming method to store and deploy them. For example, since IBRSD.conf is different in development environment and in production environment while most of other IBRSD files are the same for both environments, I store both copies of IBRSD.conf in my file repository as IBRSD.conf.dev and IBRSD.conf.prd in the same directory. When I deploy IBRSD to a production server, both IBRSD.conf.dev and IBRSD.conf.prd will be copied to the destination server. Then I will run a post deployment script to rename any file with a .prd extension to its original file name without .prd extension.
By using this renaming method, I can handle a small number of file variations while avoiding to store multiple copies of the same file in my file repository. This method also keeps the size of file repository small.
Having a small file repository means that you can back it up on a thumb drive as the files are mostly ASCII. In a catastrophic situation, you can rebuild your entire system quickly at another location with exactly the same customization you had at your original location.
How can you end up having multiple copies of the same file in your file repository? Different server roles and different environments can require different file content with the same file name. For example, .load file would be different for your PATROL cell and for your correlation cell. IBRSD.conf file would be different for your development environment and for your production environment.
Does this mean that you have to have separate file repositories for different cells and different environments? No. Since most of the files are still the same in both your development environment and your production environment, having separate file repositories can make them out of sync quickly.
Because only small percentage of customization files are different in different environments, I simply use a renaming method to store and deploy them. For example, since IBRSD.conf is different in development environment and in production environment while most of other IBRSD files are the same for both environments, I store both copies of IBRSD.conf in my file repository as IBRSD.conf.dev and IBRSD.conf.prd in the same directory. When I deploy IBRSD to a production server, both IBRSD.conf.dev and IBRSD.conf.prd will be copied to the destination server. Then I will run a post deployment script to rename any file with a .prd extension to its original file name without .prd extension.
By using this renaming method, I can handle a small number of file variations while avoiding to store multiple copies of the same file in my file repository. This method also keeps the size of file repository small.
Having a small file repository means that you can back it up on a thumb drive as the files are mostly ASCII. In a catastrophic situation, you can rebuild your entire system quickly at another location with exactly the same customization you had at your original location.
Monday, December 1, 2014
Total Cost of Ownership of BPPM - Part 8: Best practice - File repository
During BPPM implementation, you will need to create some of your own files or modify out-of-box BPPM files. These customization files include configuration files, cmd files, shell scripts, Perl scripts, KM files, PSL files, BAROC files, MRL files, etc.
One of the common practices I have seen at many organizations is to simply create new files and leave them on the server. When the same files are needed for the second server, they are copied from the first server. When the number of files becomes large, individual file copy is replaced by directory copy.
This practice certainly works in some extent, but there are several issues. For example,
1) If you make modification to some of the files while working on the third server, you will need to remember to repeat the same modification on the first and second servers. Otherwise, your files on those servers will be out of sync. A few months later, you won't remember which server contains the most up-to-date copy of your customization files.
2) When you start copying the entire directory instead of individual files, you could copy all the junk files generated during testing to the next server. A few months later, you will have a hard time to remember what those junk files are for.
3) Upgrade will be a nightmare. You will need to figure out which files you need to copy into the next version, which files you need to merge, and how many different variations of configuration/data files you need to extract out of the current version.
By using a single file repository, you can minimize even eliminate the above issues.
Pick up a shared drive as your file repository at a location where you can easily copy files to and from other servers. Keep the directory structure as identical as possible to the destination servers (such as server\etc\cellname\kb\rules\). Save only the files you created or modified in the file repository. Never copy the original BPPM out-of-box files there.
As soon as you finish testing your customization files on a server, always save a copy of most up-to-date files into your file repository. When you need to deploy files to a new server, always copy files from the file repository, not from another server. If you are not sure if a server has the most up-to-date version of your customization, simply deploy all your files again from the file repository. When it is time to upgrade, the files in your repository are the only files you you need to copy/merge into the next version.
One of the common practices I have seen at many organizations is to simply create new files and leave them on the server. When the same files are needed for the second server, they are copied from the first server. When the number of files becomes large, individual file copy is replaced by directory copy.
This practice certainly works in some extent, but there are several issues. For example,
1) If you make modification to some of the files while working on the third server, you will need to remember to repeat the same modification on the first and second servers. Otherwise, your files on those servers will be out of sync. A few months later, you won't remember which server contains the most up-to-date copy of your customization files.
2) When you start copying the entire directory instead of individual files, you could copy all the junk files generated during testing to the next server. A few months later, you will have a hard time to remember what those junk files are for.
3) Upgrade will be a nightmare. You will need to figure out which files you need to copy into the next version, which files you need to merge, and how many different variations of configuration/data files you need to extract out of the current version.
By using a single file repository, you can minimize even eliminate the above issues.
Pick up a shared drive as your file repository at a location where you can easily copy files to and from other servers. Keep the directory structure as identical as possible to the destination servers (such as server\etc\cellname\kb\rules\). Save only the files you created or modified in the file repository. Never copy the original BPPM out-of-box files there.
As soon as you finish testing your customization files on a server, always save a copy of most up-to-date files into your file repository. When you need to deploy files to a new server, always copy files from the file repository, not from another server. If you are not sure if a server has the most up-to-date version of your customization, simply deploy all your files again from the file repository. When it is time to upgrade, the files in your repository are the only files you you need to copy/merge into the next version.
Subscribe to:
Posts (Atom)