GRID 19c AIX patching with GI RELEASE UPDATE 19.6.0.0.0 fails with CLSRSC-196: ACFS driver install actions failed

Today while applying “Patch 30501910: GI RELEASE UPDATE 19.6.0.0.0” on recently upgrade GRID home from 12.2 to 19.6, I received error related ACFS ==> CLSRSC-196: ACFS driver install actions failed.

Command used for patching ==> opatchauto apply -oh /u01/app/19.6.0.0/grid /u05/ADI/patch/30501910

Detailed error is as given below:

Execution of [SIHAStartupAction] patch action failed, check log for more details. Failures:
Patch Target : test-oem01->/u01/app/19.6.0.0/grid Type[siha]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/app/19.6.0.0/grid, host: test-oem01.
Command failed: /u01/app/19.6.0.0/grid/perl/bin/perl -I/u01/app/19.6.0.0/grid/perl/lib -I/u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install -I/u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/xag /u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/roothas.pl -postpatch -norestart
Command failure output:
Using configuration parameter file: /u01/app/19.6.0.0/grid/opatchautocfg/db/dbtmp/bootstrap_test-oem01/patchwork/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/crsdata/test-oem01/crsconfig/hapatch_2020-03-17_01-46-48PM.log
2020/03/17 13:47:16 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'
2020/03/17 13:47:42 CLSRSC-196: ACFS driver install actions failed

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Tue Mar 17 13:47:43 2020
Time taken to complete the session 16 minutes, 44 seconds

opatchauto failed with error code 42

After investigating through patch log, I did see below additional details about the error:

> exec(): 0509-036 Cannot load program /usr/lib/methods/udefacfsctl.bin because of the following errors:
> 0509-150 Dependent module libhasgen12.so could not be loaded.
> 0509-022 Cannot load module libhasgen12.so.
> 0509-026 System error: A file or directory in the path name does not exist.
> ACFS-9361: Removing device 'acfsctl' failed with error code '255'.
> ACFS-9294: updating file /etc/oracledrivers.conf
> ACFS-9178: Return code = USM_FAIL
> ACFS-9177: Return from 'uninstall'
> ACFS-9305: ADVM/ACFS installation cannot proceed:
> ACFS-9306: Failed to uninstall previous installation.

So issue was pointing failed uninstall of ACFS from previous installation.

After checking on MOS, I found a note which given some idea about this behavior: AIX: ROOT.SH FAILS WITH CLSRSC-196: ACFS DRIVER INSTALL ACTIONS FAILED (Doc ID 1929899.1)

Most likely this was due to existence of older version of ACFS lib/bin files in /usr/lib/methods which prevents the new version installation. This was verified using strings command as given below:

test-oem01:/u01/app/19.6.0.0/grid/lib # strings /usr/lib/methods/udefacfsctl.bin
@(#)23 1.6 src/bos/usr/ccs/lib/libpthreads/init.c, libpth, bos610 6/21/07 15:28:59
@(#)61
1.16 src/bos/usr/ccs/lib/libc/__threads_init.c, libcthrd, bos610 8/2/07 13:09:21
Üìÿÿÿÿ
12.2.0.1.0
USM_VERSION-12.2.0.1.0.0

So we got the culprit which caused this issue. Files in /usr/lib/methods are still in older version.

Now big question was to how to fix this.

We had to deinstall older version of ACFS so to fix this:

Step 1: Copy files from new GRID home to /usr/lib/methods/

# cd /<19C GRID HOME>/usm/install/cmds/bin
# cp cfgacfsctl.bin cfgadvmctl.bin cfgadvmvol.bin defacfsctl.bin defadvmctl.bin ucfgacfsctl.bin ucfgadvmctl.bin ucfgadvmvol.bin udefacfsctl.bin udefadvmctl.bin /usr/lib/methods/

Step 2: Deinstall older ACFS

# /usr/lib/methods/ucfgacfsctl -l ofsctl
# /usr/lib/methods/ucfgadvmctl -l advmctl
# /usr/lib/methods/udefacfsctl -l ofsctl
# /usr/lib/methods/udefadvmctl -l advmctl
# /usr/sbin/rmauth -h oracle
# rmrole oracle_devmgmt
# setkst
# rm /usr/lib/drivers/oracle*
# rm /usr/lib/methods/*advm* /usr/lib/methods/*acfs*
# rm -rf /sbin/helpers/acfs
# rm /usr/sbin/acfsutil* /usr/sbin/advmutil*
# rm /sbin/acfsutil* /sbin/advmutil*

Step 3: Disable HAS

#/<19C GRID HOME>/bin/crsctl disable has

Step 4: Reboot server

Step 5: Enable HAS

#/<19C GRID HOME>/bin/crsctl eanble has

Once the above task is done, I attempted to resume my previous failed patching session using below command:

opatchauto resume -log <logfile of previous failed patching session>

This time patching did complete successfully without any errors 🙂

Hope u will find this post very useful.

Cheers

Regards,
Adityanath

DBMS_LOCK.SLEEP is now deprecated, the new SLEEP is DBMS_SESSION.SLEEP

We need sleep function many a times in our code, may be for application logic or even sometimes for monitoring purpose. We always had this available with sleep() function that resides within DBMS_LOCK ==> DBMS_LOCK.SLEEP

This was always a big security risk as granting access to DBMS_LOCK will make a way to get access to other functions/procedures within which is not always necessary.

So from 18C, Oracle comes with SLEEP function within a publicly granted package ==> DBMS_SESSION.SLEEP.

DBMS_SESSION.SLEEP

From 18C DBMS_LOCK.SLEEP is deprecated, but it is still present for backwards compatibility.

I would suggest, whoever is planning to upgrade their databases to 19C in near future, should upgrade their PL/SQL codes to use DBMS_SESSION.SLEEP instead of DBMS_LOCK.SLEEP. Also one should make sure to revoke any grants to the DBMS_LOCK package where they were intended to give access to only SLEEP procedure.

References:

ER 23557076 : PUBLIC SLEEP FUNCTION

Hope u will find this post very useful. 🙂

Cheers

Regards,

Adityanath

 

SQL*Plus command line history on UNIX platform

One of the best feature I came across recently. Oracle SQL*Plus is now support command line history similar to UNIX “history” command. This feature is available from Oracle 12.2.

Sample example is given below:

You can refer below link for more details:

SQL*Plus HISTORY

Hope u will find this post very useful. 🙂

Cheers

Regards,
Adityanath

Why should people attend AIOUG events :-)

Dear Connections,

I was chatting to one of my dear friend yesterday during Sangam19 and he turned to me and asked “Why should people attend any AIOUG events”. It made me pause to think for a moment. Any type of such events is going to take up your time and will usually cost a decent amount of money. In addition, you’re taking time away from work & its not for vacation.

This post is to share my personal top reasons as to why people should attend AIOUG events:

Expand your professional network:

These events provide great opportunity to network with like minded people and industry peers.It is always helpful to have a healthy professional network. These events bring together people from all different geographical areas under single roof. Also this helps you meeting with people in your field that you haven’t connected in a while.

Expand your knowledge:

No matter how experienced you are at your field, everyone can learn. These is the best opportunity to hear from all Industry experts in person. You will meet all Oracle Gurus wherein they will share their experiences, their learning on the latest technologies. Also you can get guidance from them, also you can clear your doubts.

Learn beyond your skills:

When you are working for the same company for the years, your work & knowledge can become monotonous. Most of the times due to busy work schedule, its very difficult keep yourself up with the technology changes and advancements. These events can help you to know about newest technologies in the market also you can plan your careers accordingly.

To present yourself:

These events are the best opportunities wherein you can market yourself, share your ideas, knowledge. Here you can meet someone who can influence your professional career dramatically. You can develop a reputation as an expert to your peers.

Get certified:

All AIOUG events, provide your considerable discounts on all Oracle certifications. Especially in during Sangam19 Test Fest, people completed their certifications almost in free of cost.

Have Fun:

Last but of course not the least is the fun part. These events are fully filled with fun activities. These events can be a much needed break from your day to day life.

I am happy share some of the best pics from Sangam19:

 

Hope u will find this post very useful. 🙂

Cheers

Regards,
Adityanath

 

OEM agent version 13c installation on AIX fails with OUI-10039:Unable to access the inventory /u01/app/oraInventory on this system.

Hello Readers,

Few days ago, I was installing OEM agent version 13.3.0.0.0 on my DEV box with OS AIX version 7.2. I was using silent method using agentDeploy.sh.

Previously I had installed it successfully on multiple machines without any issues but this one failed with below error:

java.io.IOException: OUI-10039:Unable to access the inventory /u01/app/oraInventory on this system. Please ensure you have the proper permissions to read/write/search the inventory.

I tried to see if there are any permissions issue on folder /u01/app/oraInventory, I found mentioned directory was not present. So this error is expected one. But why Oracle is searching inventory at incorrect location???

When I dug into associated log file, I found more details about these errors.

2019-10-07 12:20:48,465 WARNING [34] oracle.sysman.oii.oiip.oiipg.OiipgPropertyLoader - The inventory pointer location /var/opt/oracle/oraInst.loc is either not readable or does not exist
2019-10-07 12:20:48,475 INFO [34] oracle.sysman.nextgen.utils.NextGenInventoryUtil - Setting default inventory location to: '/u01/app/oraInventory'
2019-10-07 12:20:48,475 WARNING [34] oracle.sysman.oii.oiip.oiipg.OiipgPropertyLoader - The inventory pointer location /var/opt/oracle/oraInst.loc is either not readable or does not exist
2019-10-07 12:20:48,475 WARNING [34] oracle.sysman.oii.oiip.oiipg.OiipgPropertyLoader - The inventory pointer location /var/opt/oracle/oraInst.loc is either not readable or does not exist
2019-10-07 12:20:48,477 SEVERE [34] oracle.sysman.oii.oiii.OiiiInstallAreaControl - OUI-10039:Unable to access the inventory /u01/app/oraInventory on this system. Please ensure you have the proper permissions to read/write/search the inventory.
2019-10-07 12:20:48,477 SEVERE [34] oracle.sysman.nextgen.impl.NextGenInstallerImpl - java.io.IOException: OUI-10039:Unable to access the inventory /u01/app/oraInventory on this system. Please ensure you have the proper permissions to read/write/search the inventory.

So basically, Oracle tries to check oraInst.loc under folder /var/opt/oracle & if it doesn’t find any, then it sets default inventory location to ‘/u01/app/oraInventory’.

I feel, This is somewhat agent software BUG, as loaction of oraInst.loc in AIX is ‘/etc’ not ‘/var/opt/oracle’.

Now there are two questions. First one, how to resolve this & second one, why my other installations went successful.

Reason behind other installation to be successful was Oracle indeed find oraInventory at its default location. So whenever oraInventory is located at ‘/u01/app/oraInventory’, you wont face this issue.

Now how to resolve this. You can always create softlink oraInst.loc to “/var/opt/oracle”.

Steps are given below:

Login using root:
1. mkdir -p /var/opt/oracle/
2. cd /var/opt/oracle/
3. ln -s /etc/oraInst.loc oraInst.loc
4. ls -lrt oraInst.loc

Once you perform above steps, you will be install OEM agent successfully.

Hope u will find this post very useful. 🙂

Cheers

Regards,
Adityanath