Today while applying “Patch 30501910: GI RELEASE UPDATE 18.104.22.168.0” on recently upgrade GRID home from 12.2 to 19.6, I received error related ACFS ==> CLSRSC-196: ACFS driver install actions failed.
Command used for patching ==> opatchauto apply -oh /u01/app/22.214.171.124/grid /u05/ADI/patch/30501910
Detailed error is as given below:
Execution of [SIHAStartupAction] patch action failed, check log for more details. Failures: Patch Target : test-oem01->/u01/app/126.96.36.199/grid Type[siha] Details: [ ---------------------------Patching Failed--------------------------------- Command execution failed during patching in home: /u01/app/188.8.131.52/grid, host: test-oem01. Command failed: /u01/app/184.108.40.206/grid/perl/bin/perl -I/u01/app/220.127.116.11/grid/perl/lib -I/u01/app/18.104.22.168/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install -I/u01/app/22.214.171.124/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/xag /u01/app/126.96.36.199/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/roothas.pl -postpatch -norestart Command failure output: Using configuration parameter file: /u01/app/188.8.131.52/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/crsconfig_params The log of current session can be found at: /u01/app/crsdata/test-oem01/crsconfig/hapatch_2020-03-17_01-46-48PM.log 2020/03/17 13:47:16 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab' 2020/03/17 13:47:42 CLSRSC-196: ACFS driver install actions failed After fixing the cause of failure Run opatchauto resume ] OPATCHAUTO-68061: The orchestration engine failed. OPATCHAUTO-68061: The orchestration engine failed with return code 1 OPATCHAUTO-68061: Check the log for more details. OPatchAuto failed. OPatchauto session completed at Tue Mar 17 13:47:43 2020 Time taken to complete the session 16 minutes, 44 seconds opatchauto failed with error code 42
After investigating through patch log, I did see below additional details about the error:
> exec(): 0509-036 Cannot load program /usr/lib/methods/udefacfsctl.bin because of the following errors: > 0509-150 Dependent module libhasgen12.so could not be loaded. > 0509-022 Cannot load module libhasgen12.so. > 0509-026 System error: A file or directory in the path name does not exist. > ACFS-9361: Removing device 'acfsctl' failed with error code '255'. > ACFS-9294: updating file /etc/oracledrivers.conf > ACFS-9178: Return code = USM_FAIL > ACFS-9177: Return from 'uninstall' > ACFS-9305: ADVM/ACFS installation cannot proceed: > ACFS-9306: Failed to uninstall previous installation.
So issue was pointing failed uninstall of ACFS from previous installation.
After checking on MOS, I found a note which given some idea about this behavior: AIX: ROOT.SH FAILS WITH CLSRSC-196: ACFS DRIVER INSTALL ACTIONS FAILED (Doc ID 1929899.1)
Most likely this was due to existence of older version of ACFS lib/bin files in /usr/lib/methods which prevents the new version installation. This was verified using strings command as given below:
test-oem01:/u01/app/184.108.40.206/grid/lib # strings /usr/lib/methods/udefacfsctl.bin @(#)23 1.6 src/bos/usr/ccs/lib/libpthreads/init.c, libpth, bos610 6/21/07 15:28:59 @(#)61 1.16 src/bos/usr/ccs/lib/libc/__threads_init.c, libcthrd, bos610 8/2/07 13:09:21 Üìÿÿÿÿ 220.127.116.11.0 USM_VERSION-18.104.22.168.0.0
So we got the culprit which caused this issue. Files in /usr/lib/methods are still in older version.
Now big question was to how to fix this.
We had to deinstall older version of ACFS so to fix this:
Step 1: Copy files from new GRID home to /usr/lib/methods/
# cd /<19C GRID HOME>/usm/install/cmds/bin
# cp cfgacfsctl.bin cfgadvmctl.bin cfgadvmvol.bin defacfsctl.bin defadvmctl.bin ucfgacfsctl.bin ucfgadvmctl.bin ucfgadvmvol.bin udefacfsctl.bin udefadvmctl.bin /usr/lib/methods/
Step 2: Deinstall older ACFS
# /usr/lib/methods/ucfgacfsctl -l ofsctl
# /usr/lib/methods/ucfgadvmctl -l advmctl
# /usr/lib/methods/udefacfsctl -l ofsctl
# /usr/lib/methods/udefadvmctl -l advmctl
# /usr/sbin/rmauth -h oracle
# rmrole oracle_devmgmt
# rm /usr/lib/drivers/oracle*
# rm /usr/lib/methods/*advm* /usr/lib/methods/*acfs*
# rm -rf /sbin/helpers/acfs
# rm /usr/sbin/acfsutil* /usr/sbin/advmutil*
# rm /sbin/acfsutil* /sbin/advmutil*
Step 3: Disable HAS
#/<19C GRID HOME>/bin/crsctl disable has
Step 4: Reboot server
Step 5: Enable HAS
#/<19C GRID HOME>/bin/crsctl eanble has
Once the above task is done, I attempted to resume my previous failed patching session using below command:
opatchauto resume -log <logfile of previous failed patching session>
This time patching did complete successfully without any errors 🙂
Hope u will find this post very useful.