Today while applying “Patch 30501910: GI RELEASE UPDATE 19.6.0.0.0” on recently upgrade GRID home from 12.2 to 19.6, I received error related ACFS ==> CLSRSC-196: ACFS driver install actions failed.

Command used for patching ==> opatchauto apply -oh /u01/app/19.6.0.0/grid /u05/ADI/patch/30501910

Detailed error is as given below:

Execution of [SIHAStartupAction] patch action failed, check log for more details. Failures:
Patch Target : test-oem01->/u01/app/19.6.0.0/grid Type[siha]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.6.0.0/grid, host: test-oem01.
Command failed: /u01/app/19.6.0.0/grid/perl/bin/perl -I/u01/app/19.6.0.0/grid/perl/lib -I/u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install -I/u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/xag /u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/roothas.pl -postpatch -norestart
Command failure output:
Using configuration parameter file: /u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/crsdata/test-oem01/crsconfig/hapatch_2020-03-17_01-46-48PM.log
2020/03/17 13:47:16 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'
2020/03/17 13:47:42 CLSRSC-196: ACFS driver install actions failed

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Tue Mar 17 13:47:43 2020
Time taken to complete the session 16 minutes, 44 seconds

opatchauto failed with error code 42

After investigating through patch log, I did see below additional details about the error:

> exec(): 0509-036 Cannot load program /usr/lib/methods/udefacfsctl.bin because of the following errors:
> 0509-150 Dependent module libhasgen12.so could not be loaded.
> 0509-022 Cannot load module libhasgen12.so.
> 0509-026 System error: A file or directory in the path name does not exist.
> ACFS-9361: Removing device 'acfsctl' failed with error code '255'.
> ACFS-9294: updating file /etc/oracledrivers.conf
> ACFS-9178: Return code = USM_FAIL
> ACFS-9177: Return from 'uninstall'
> ACFS-9305: ADVM/ACFS installation cannot proceed:
> ACFS-9306: Failed to uninstall previous installation.

So issue was pointing failed uninstall of ACFS from previous installation.

After checking on MOS, I found a note which given some idea about this behavior: AIX: ROOT.SH FAILS WITH CLSRSC-196: ACFS DRIVER INSTALL ACTIONS FAILED (Doc ID 1929899.1)

Most likely this was due to existence of older version of ACFS lib/bin files in /usr/lib/methods which prevents the new version installation. This was verified using strings command as given below:

test-oem01:/u01/app/19.6.0.0/grid/lib # strings /usr/lib/methods/udefacfsctl.bin
@(#)23 1.6 src/bos/usr/ccs/lib/libpthreads/init.c, libpth, bos610 6/21/07 15:28:59
@(#)61
1.16 src/bos/usr/ccs/lib/libc/__threads_init.c, libcthrd, bos610 8/2/07 13:09:21
Üìÿÿÿÿ
12.2.0.1.0
USM_VERSION-12.2.0.1.0.0

So we got the culprit which caused this issue. Files in /usr/lib/methods are still in older version.

Now big question was to how to fix this.

We had to deinstall older version of ACFS so to fix this:

Step 1: Copy files from new GRID home to /usr/lib/methods/

# cd /<19C GRID HOME>/usm/install/cmds/bin
# cp cfgacfsctl.bin cfgadvmctl.bin cfgadvmvol.bin defacfsctl.bin defadvmctl.bin ucfgacfsctl.bin ucfgadvmctl.bin ucfgadvmvol.bin udefacfsctl.bin udefadvmctl.bin /usr/lib/methods/

Step 2: Deinstall older ACFS

# /usr/lib/methods/ucfgacfsctl -l ofsctl
# /usr/lib/methods/ucfgadvmctl -l advmctl
# /usr/lib/methods/udefacfsctl -l ofsctl
# /usr/lib/methods/udefadvmctl -l advmctl
# /usr/sbin/rmauth -h oracle
# rmrole oracle_devmgmt
# setkst
# rm /usr/lib/drivers/oracle*
# rm /usr/lib/methods/*advm* /usr/lib/methods/*acfs*
# rm -rf /sbin/helpers/acfs
# rm /usr/sbin/acfsutil* /usr/sbin/advmutil*
# rm /sbin/acfsutil* /sbin/advmutil*

Step 3: Disable HAS

#/<19C GRID HOME>/bin/crsctl disable has

Step 4: Reboot server

Step 5: Enable HAS

#/<19C GRID HOME>/bin/crsctl eanble has

Once the above task is done, I attempted to resume my previous failed patching session using below command:

opatchauto resume -log <logfile of previous failed patching session>

This time patching did complete successfully without any errors 🙂

Hope u will find this post very useful.

Cheers

Regards,
Adityanath