GRID 19c AIX patching with GI RELEASE UPDATE 19.6.0.0.0 fails with CLSRSC-196: ACFS driver install actions failed

Today while applying “Patch 30501910: GI RELEASE UPDATE 19.6.0.0.0” on recently upgrade GRID home from 12.2 to 19.6, I received error related ACFS ==> CLSRSC-196: ACFS driver install actions failed.

Command used for patching ==> opatchauto apply -oh /u01/app/19.6.0.0/grid /u05/ADI/patch/30501910

Detailed error is as given below:


Execution of [SIHAStartupAction] patch action failed, check log for more details. Failures:
Patch Target : test-oem01->/u01/app/19.6.0.0/grid Type[siha]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.6.0.0/grid, host: test-oem01.
Command failed: /u01/app/19.6.0.0/grid/perl/bin/perl -I/u01/app/19.6.0.0/grid/perl/lib -I/u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install -I/u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/xag /u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/roothas.pl -postpatch -norestart
Command failure output:
Using configuration parameter file: /u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/crsdata/test-oem01/crsconfig/hapatch_2020-03-17_01-46-48PM.log
2020/03/17 13:47:16 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'
2020/03/17 13:47:42 CLSRSC-196: ACFS driver install actions failed

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Tue Mar 17 13:47:43 2020
Time taken to complete the session 16 minutes, 44 seconds

opatchauto failed with error code 42

After investigating through patch log, I did see below additional details about the error:


> exec(): 0509-036 Cannot load program /usr/lib/methods/udefacfsctl.bin because of the following errors:
> 0509-150 Dependent module libhasgen12.so could not be loaded.
> 0509-022 Cannot load module libhasgen12.so.
> 0509-026 System error: A file or directory in the path name does not exist.
> ACFS-9361: Removing device 'acfsctl' failed with error code '255'.
> ACFS-9294: updating file /etc/oracledrivers.conf
> ACFS-9178: Return code = USM_FAIL
> ACFS-9177: Return from 'uninstall'
> ACFS-9305: ADVM/ACFS installation cannot proceed:
> ACFS-9306: Failed to uninstall previous installation.

So issue was pointing failed uninstall of ACFS from previous installation.

After checking on MOS, I found a note which given some idea about this behavior: AIX: ROOT.SH FAILS WITH CLSRSC-196: ACFS DRIVER INSTALL ACTIONS FAILED (Doc ID 1929899.1)

Most likely this was due to existence of older version of ACFS lib/bin files in /usr/lib/methods which prevents the new version installation. This was verified using strings command as given below:


test-oem01:/u01/app/19.6.0.0/grid/lib # strings /usr/lib/methods/udefacfsctl.bin
@(#)23 1.6 src/bos/usr/ccs/lib/libpthreads/init.c, libpth, bos610 6/21/07 15:28:59
@(#)61
1.16 src/bos/usr/ccs/lib/libc/__threads_init.c, libcthrd, bos610 8/2/07 13:09:21
Üìÿÿÿÿ
12.2.0.1.0
USM_VERSION-12.2.0.1.0.0

So we got the culprit which caused this issue. Files in /usr/lib/methods are still in older version.

Now big question was to how to fix this.

We had to deinstall older version of ACFS so to fix this:


Step 1: Copy files from new GRID home to /usr/lib/methods/

# cd /<19C GRID HOME>/usm/install/cmds/bin
# cp cfgacfsctl.bin cfgadvmctl.bin cfgadvmvol.bin defacfsctl.bin defadvmctl.bin ucfgacfsctl.bin ucfgadvmctl.bin ucfgadvmvol.bin udefacfsctl.bin udefadvmctl.bin /usr/lib/methods/

Step 2: Deinstall older ACFS

# /usr/lib/methods/ucfgacfsctl -l ofsctl
# /usr/lib/methods/ucfgadvmctl -l advmctl
# /usr/lib/methods/udefacfsctl -l ofsctl
# /usr/lib/methods/udefadvmctl -l advmctl
# /usr/sbin/rmauth -h oracle
# rmrole oracle_devmgmt
# setkst
# rm /usr/lib/drivers/oracle*
# rm /usr/lib/methods/*advm* /usr/lib/methods/*acfs*
# rm -rf /sbin/helpers/acfs
# rm /usr/sbin/acfsutil* /usr/sbin/advmutil*
# rm /sbin/acfsutil* /sbin/advmutil*

Step 3: Disable HAS

#/<19C GRID HOME>/bin/crsctl disable has

Step 4: Reboot server

Step 5: Enable HAS

#/<19C GRID HOME>/bin/crsctl eanble has


Once the above task is done, I attempted to resume my previous failed patching session using below command:

opatchauto resume -log <logfile of previous failed patching session>

This time patching did complete successfully without any errors 🙂

Hope u will find this post very useful.

Cheers

Regards,
Adityanath

12 replies

  1. Like!! Really appreciate you sharing this blog post.Really thank you! Keep writing.

  2. These are actually great ideas in concerning blogging.

  3. After going over a few of the blog posts on your
    web page, I honestly like your technique of writing a blog.
    I added it to my bookmark webpage list and will be checking back
    soon. Please check out my web site as well and let me know how you
    feel.

  4. Hello very cool web site!! Guy .. Beautiful .. Superb .. I’ll bookmark your web site and take the feeds I’m glad to search out so many useful information here within the publish, we’d like develop more strategies on this regard, thank you for sharing. . . . . .

  5. I’ve been absent for some time, but now I remember why I used to love this site. Thank you, I will try and check back more frequently. How frequently you update your website?

  6. Hey there! I’m at work surfing around your blog from my new
    iphone! Just wanted to say I love reading your blog and look forward to all your posts!
    Keep up the fantastic work!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s