Experience share on how to perform trouble shooting – a CRM Central Instance down problem

As my last blog described, I met the problem which had the following screenshot.

Facing such a problem, how to perform the trouble shooting?

 

First, make the time table of the trouble since it happens for the first time.

In this case, as I checked, the time table of the problem is as following:

9.14.08      CRM System installed successfully

9.16.08 – 9.17.08 Some system post-installations and profile modification were performed.

9.20.08      The server where the CSH were installed was shut down by accident.

9.22.08     After restarting the server, the SAP system were found dead and the above screenshot appeared.

 

Second, roughly analysis the area the problem possibly covers and try to narrow down the possible causes.

An instance cannot start up and the database can't show up in the SAPMMC, so we can make sure that the most big three layers: SAP System, the SAPMMC and the DB2 database.

And now, I should NARROW DOWN the problem!

To make sure whether it's the SAPMMC's problem, I decide to figure out if I can start the SAP system via command line or in some other way.

And soon I got the right way to start a sap system via command line and confirmed the way on another system, it worked~!

The cmd script to start a sap instance: (in the E://usr/sap/csh/sys/exe/uc/ntamd64/ )sapstart pf=<START_PROFILE_OF_THE_INSTANCE>

The cmd script to stop a sap instance: (the same directory as above) stopsap <SID> <SYSTEM_NO> <HOSTNAME> pf=<INSTANCE_PROFILE_OF_THE_INSTANCE>

Here we should be aware of the profile type and the way we wrote the script, I tried and some other ways but failed.

These commands ran successfully on my server and the instances in the SAPMMC seem standstill, which means, I still cannot decide whether the problem is none business with SAPMMC. And also apparently, the problem of the CRM system exists.

So I rechecked the start logs of the CI instance and found that in the startsap.log, I could only find such lines,

This suggests that the sapstart program never call up the message server process, the igs server or the disp+work process.

So I compare the START_<InstanceName+NR>_<Hostname> file of CSH and ESH and find the following part of ESH have no responding part in the CSH's start profile.

After I added something like this into the CSH's startup profile, the central instance finally started although still with the disp+work process down.

 

After what I did above, the problem was narrowed again. The log of the instance showed an "Invalid profile parameter of DBMS_TYPE …" error happened in each work process's log file. With the help of colleagues from database team, the problem was finally solved.

 

The End.