InstallFixpack command fails with "ERROR: The installFixPack command detected some DB2 libraries are still loaded in memory"

Troubleshooting

Problem

The installFixpack command fails indicating that there are DB2 libraries still loaded in memory

Symptom

You will see error output similar to the following
----------------------------------------------------------------------------------------------------------------
ERROR: The installFixPack command detected some DB2 libraries are still loaded in memory and some applications might still be running in the current installation copy. All applications must be stopped. See the fix pack readme for pre-installation instructions, and re-run the installFixPack command.

Alternatively, to override automatic checking, you can re-issue the installFixPack command with the '-f db2lib' parameter.

Note: If you re-issue the installFixPack command with the '-f db2lib' parameter, after the DB2 instances are updated, some applications might not work properly and might need to be restarted to function properly against the updated DB2 instance.

Preparing the system :.......Failure

Environment

This technote is valid for AIX and Linux operating systems.

Diagnosing The Problem

This error is caused by a process that is still holding one of the db2 libraries in memory. In order to help determine if there is an issue with one of the db2 libraries you can run as 'root' the db2prechk command.

Run the following:

<INSTALL_PATH>/instance/db2prechk -b /opt/ibm/db2/V9.7 -d;echo $?
--------------------------------------------------------------------------------
If 0 is returned then no db2 libraries were found.
You can workaround the issue by running the installFixpack with -f db2lib option

If 1 is returned then you will also see in the list generated by db2prechk the libraries that are 'IN USE'

Similar to the following
/opt/ibm/db2/V9.7/lib32/libdb2locale.so: NOT IN USE
/opt/ibm/db2/V9.7/lib32/libdb2locale.so.1: NOT IN USE
/opt/ibm/db2/V9.7/lib32/libdb2osse_db2.so: NOT IN USE
/opt/ibm/db2/V9.7/lib32/libdb2osse_db2.so.1: NOT IN USE
/opt/ibm/db2/V9.7/lib32/libdb2osse.so: NOT IN USE
/opt/ibm/db2/V9.7/lib32/libdb2osse.so.1: NOT IN USE
/opt/ibm/db2/V9.7/lib64/libdb2locale.so: IN USE
1

Resolving The Problem

On AIX you can use genld to find what process is using the library

Run:
/usr/bin/genld -l | grep -p db2

You'll see something similar to the following

In this example db2fmcd is holding the library

Proc_pid: 520362 Proc_name: db2fmcd
100000000 13f72 db2fmcd
9fffffff0000000 c25a /usr/ccs/bin/usla64
90000000125b100 11680 /opt/IBM/db2/V9.5/lib64/libgcffmcmd.a[shr_64.o]
9000000006cd000 18d6f /usr/lib/libiconv.a[shr4_64.o]
9000000006be880 dd8a /usr/lib/libi18n.a[shr_64.o]
9000000006b5000 424d /usr/lib/nls/loc/en_US__64
9000000006b3c38 a3b /usr/lib/libC.a[shr3_64.o]
9000000006b1a40 a6a /usr/lib/libC.a[shr2_64.o]
900000000611380 af3 /usr/lib/libcrypt.a[shr_64.o]
90000000062ee00 19c09 /usr/lib/libC.a[shrcore_64.o]
9000000006126c0 1ad1b /usr/lib/libC.a[ansicore_64.o]
900000000ac5940 5bff /opt/IBM/db2/V9.5/lib64/libdb2install.a[shr_64.o]
900000000a86b00 33710 /opt/IBM/db2/V9.5/lib64/libdb2genreg.a[shr_64.o]
900000001256100 4eec /opt/IBM/db2/V9.5/lib64/libgcf.a[shr_64.o]
9000000007075c0 37df0c /opt/IBM/db2/V9.5/lib64/libdb2osse.a[shr_64.o]
90000000047bc00 17cf14 /usr/lib/libC.a[ansi_64.o]
90000000065e000 3e1a8 /usr/lib/libpthreads.a[shr_xpg5_64.o]
900000000000900 392520 /usr/lib/threads/libc.a[shr_64.o]
900000000393e40 21e8e /usr/lib/libC.a[shr_64.o]

-----------------------------------------------------------------------------------------------------
On Linux you can use lsof +D <install_dir> or lsof | grep db2

Example where <install_dir> is /opt/ibm/db2/V9.7

lsof +D /opt/ibm/db2/V9.7

You'll see something similar to the output below. In this example it is the db2bp process holding the library:

COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
db2bp 7309 db2inst1 txt REG 253,14 1406599 426900 /opt/ibm/db2/V9.7/bin/db2bp
db2bp 7309 db2inst1 mem REG 253,14 22575008 606238 /opt/ibm/db2/V9.7/lib64/libdb2.so.1
db2bp 7309 db2inst1 mem REG 253,14 6813854 442567 /opt/ibm/db2/V9.7/lib64/libdb2g11n.so.1

Clean up the above processes and rerun the db2prechk command to verify that it returns 0.

Then retry the normal installFixPack command. If it still fails with the same error and there are no libraries loaded per the db2prechk output then you can run the installFixpack with the -f db2lib command to bypass the library check.