Today while applying “Patch 30501910: GI RELEASE UPDATE 126.96.36.199.0” on recently upgrade GRID home from 12.2 to 19.6, I received error related ACFS ==> CLSRSC-196: ACFS driver install actions failed.
Command used for patching ==> opatchauto apply -oh /u01/app/188.8.131.52/grid /u05/ADI/patch/30501910
Detailed error is as given below:
Execution of [SIHAStartupAction] patch action failed, check log for more details. Failures: Patch Target : test-oem01->/u01/app/184.108.40.206/grid Type[siha] Details: [ ---------------------------Patching Failed--------------------------------- Command execution failed during patching in home: /u01/app/220.127.116.11/grid, host: test-oem01. Command failed: /u01/app/18.104.22.168/grid/perl/bin/perl -I/u01/app/22.214.171.124/grid/perl/lib -I/u01/app/126.96.36.199/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install -I/u01/app/188.8.131.52/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/xag /u01/app/184.108.40.206/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/roothas.pl -postpatch -norestart Command failure output: Using configuration parameter file: /u01/app/220.127.116.11/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/crsconfig_params The log of current session can be found at: /u01/app/crsdata/test-oem01/crsconfig/hapatch_2020-03-17_01-46-48PM.log 2020/03/17 13:47:16 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab' 2020/03/17 13:47:42 CLSRSC-196: ACFS driver install actions failed After fixing the cause of failure Run opatchauto resume ] OPATCHAUTO-68061: The orchestration engine failed. OPATCHAUTO-68061: The orchestration engine failed with return code 1 OPATCHAUTO-68061: Check the log for more details. OPatchAuto failed. OPatchauto session completed at Tue Mar 17 13:47:43 2020 Time taken to complete the session 16 minutes, 44 seconds opatchauto failed with error code 42
After investigating through patch log, I did see below additional details about the error:
> exec(): 0509-036 Cannot load program /usr/lib/methods/udefacfsctl.bin because of the following errors: > 0509-150 Dependent module libhasgen12.so could not be loaded. > 0509-022 Cannot load module libhasgen12.so. > 0509-026 System error: A file or directory in the path name does not exist. > ACFS-9361: Removing device 'acfsctl' failed with error code '255'. > ACFS-9294: updating file /etc/oracledrivers.conf > ACFS-9178: Return code = USM_FAIL > ACFS-9177: Return from 'uninstall' > ACFS-9305: ADVM/ACFS installation cannot proceed: > ACFS-9306: Failed to uninstall previous installation.
So issue was pointing failed uninstall of ACFS from previous installation.
After checking on MOS, I found a note which given some idea about this behavior: AIX: ROOT.SH FAILS WITH CLSRSC-196: ACFS DRIVER INSTALL ACTIONS FAILED (Doc ID 1929899.1)
Most likely this was due to existence of older version of ACFS lib/bin files in /usr/lib/methods which prevents the new version installation. This was verified using strings command as given below:
test-oem01:/u01/app/18.104.22.168/grid/lib # strings /usr/lib/methods/udefacfsctl.bin @(#)23 1.6 src/bos/usr/ccs/lib/libpthreads/init.c, libpth, bos610 6/21/07 15:28:59 @(#)61 1.16 src/bos/usr/ccs/lib/libc/__threads_init.c, libcthrd, bos610 8/2/07 13:09:21 Üìÿÿÿÿ 22.214.171.124.0 USM_VERSION-126.96.36.199.0.0
So we got the culprit which caused this issue. Files in /usr/lib/methods are still in older version.
Now big question was to how to fix this.
We had to deinstall older version of ACFS so to fix this:
Step 1: Copy files from new GRID home to /usr/lib/methods/
# cd /<19C GRID HOME>/usm/install/cmds/bin
# cp cfgacfsctl.bin cfgadvmctl.bin cfgadvmvol.bin defacfsctl.bin defadvmctl.bin ucfgacfsctl.bin ucfgadvmctl.bin ucfgadvmvol.bin udefacfsctl.bin udefadvmctl.bin /usr/lib/methods/
Step 2: Deinstall older ACFS
# /usr/lib/methods/ucfgacfsctl -l ofsctl
# /usr/lib/methods/ucfgadvmctl -l advmctl
# /usr/lib/methods/udefacfsctl -l ofsctl
# /usr/lib/methods/udefadvmctl -l advmctl
# /usr/sbin/rmauth -h oracle
# rmrole oracle_devmgmt
# rm /usr/lib/drivers/oracle*
# rm /usr/lib/methods/*advm* /usr/lib/methods/*acfs*
# rm -rf /sbin/helpers/acfs
# rm /usr/sbin/acfsutil* /usr/sbin/advmutil*
# rm /sbin/acfsutil* /sbin/advmutil*
Step 3: Disable HAS
#/<19C GRID HOME>/bin/crsctl disable has
Step 4: Reboot server
Step 5: Enable HAS
#/<19C GRID HOME>/bin/crsctl eanble has
Once the above task is done, I attempted to resume my previous failed patching session using below command:
opatchauto resume -log <logfile of previous failed patching session>
This time patching did complete successfully without any errors 🙂
Hope u will find this post very useful.
Categories: 19c, GRID patching/upgrade, Installation, Upgrade
well-done Adi .
Thanks so much Dibs!!!
Like!! Really appreciate you sharing this blog post.Really thank you! Keep writing.
These are actually great ideas in concerning blogging.
Thanks so much for the blog post.
bookmarked!!, I like your blog!
some really interesting information, well written and broadly speaking user pleasant.
After going over a few of the blog posts on your
web page, I honestly like your technique of writing a blog.
I added it to my bookmark webpage list and will be checking back
soon. Please check out my web site as well and let me know how you
Hello very cool web site!! Guy .. Beautiful .. Superb .. I’ll bookmark your web site and take the feeds I’m glad to search out so many useful information here within the publish, we’d like develop more strategies on this regard, thank you for sharing. . . . . .
I’ve been absent for some time, but now I remember why I used to love this site. Thank you, I will try and check back more frequently. How frequently you update your website?
Hey there! I’m at work surfing around your blog from my new
iphone! Just wanted to say I love reading your blog and look forward to all your posts!
Keep up the fantastic work!
Appreciate this post. Let me try it out.