Skip to main content
Broadpin

First Experience with E-Business Suite 12.2.13 / ECC V11 Upgrade

This blog post was written by Johannes Michler (Chief Technology Officer (CTO) @ Broadpin & Oracle Ace Director).

Symptoms of Issue while applying the patch

While/after applying the main 12.2.13 patch (34776655) unfortunately the instance broke down terribly. The issues that we were seeing were:

  • Oracle Forms could no longer be opened. The following error was shown:

Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.

  • Enterprise Command Centers can no longer be used. In the eccManaged Server we see logs as follows:

Caused by: java.lang.RuntimeException: java.sql.SQLException: ORA-01017: invalid username/password; logon denied

at oracle.apps.fnd.security.AppsConnectionManager.makeGuestConnection(AppsConnectionManager.java:920)
at oracle.apps.fnd.security.DBConnObj.getLabelledConnection(DBConnObj.java:298)

  • The same error can also be seen with trivial dbc-based connection tests

  • Since all that suggests there are issues with the APPS-Password, I then tried to change the APPS/APPLSYS/APPS_NE password using AFPASS. That gives:

[oracle@master230703app01 ~]$ date; AFPASSWD -s APPLSYS; date
Wed Nov 22 13:11:02 CET 2023
Enter the ORACLE password of Application Object Library 'APPSUSER':
Connected successfully to APPS.
Error in password verification for APPS
AFPASSWD completed with errors.
Wed Nov 22 13:11:06 CET 2023

  • Using the “older” tool to change the passwords FNDCPASS was similar:

oracle@master230703app01 ~]$ FNDCPASS APPS/AbCdEfG 0 Y EBS_SYSTEM/Secret1234 SYSTEM APPLSYS MYNEWPWD
Log filename : L858096.log

Report filename : O858096.out
[oracle@master230703app01 ~]$ cat L858096.log
+---------------------------------------------------------------------------+
Application Object Library: Version : 12.2

Copyright (c) 1998, 2013, Oracle and/or its affiliates. All rights reserved.

FNDCPASS:
+---------------------------------------------------------------------------+

Current system time is 21-NOV-2023 00:19:40

+---------------------------------------------------------------------------+

Arguments

FNDCPASS EBS_SYSTEM/***** SYSTEM APPLSYS *****
+----------------------------------------------------------------------------+
Working...
Error in password verification for APPS

+---------------------------------------------------------------------------+
Concurrent request completed
Current system time is 21-NOV-2023 00:19:40

+---------------------------------------------------------------------------+

  • Checking if the E-Business Suite Concurrent Managers are up also fails:

[oracle@master230703app01 ~]$ adcmctl.sh status

You are running adcmctl.sh version 120.19.12020000.7

Enter the APPS username : apps

Enter the APPS password :
Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.
Internal Concurrent Manager status could not be determined.

adcmctl.sh: exiting with status 0

adcmctl.sh: check the logfile /u01/install/APPS/fs2/inst/apps/MAST_mastebsapp01/logs/appl/admin/log/adcmctl.txt for more information ...

All those issues suggest, that something is broken with the Vault.

Cause

All those issues started even during the application of the patch; so even before the cutover phase the “run” edition of the E-Business Suite got broken.

A lot of analysis together with the E-Business Suite development team at Oracle revealed, that the problem is related with us having:

  1. The sec_case_sensitive_logon database parameter is set to FALSE. So this means, that password checking on the DB is not done case sensitive

  2. However, the APPS/APPLSYS/APPS_NE password on that instance has previously been set to a mixed case Password (aBcDeFgH instead of ABCDEFGH). While this is not a problem in normal 12.2.12 and AD/TXK 15 operation, somehow the patch stumbles over this.

Be aware: We've been "rather lucky" that the instance were we first realized this was not PROD. Even if you do all your testing first on a Dev/Test instance you might hit this issue only on PROD. Just think about having APPS as the APPS-Password on Dev/Test (I've heard of that ? ) and then having something FaNcY as the PROD password. Then the Dev/Test works perfect but on PROD you break the PROD even before you start your planned downtime for the ADOP Cutover.

Solution A: Fix before upgrade

If you’ve got a valid backup that you can easily restore probably the easiest approach to fix this is:

  • Before applying the 12.2.13 patch change the password to something “all uppercase” (e.g. MYPWD) using AFPASSWD. See https://docs.oracle.com/cd/E26401_01/doc.122/e22954/T202991T174305.htm#522325 for all details.

  • Optionally ( I did this, I don’t think though that it is necessary) enable sec_case_sensitive_logon to TRUE and (further) optionally change the password back to something with mixed case for better security.

  • Only then apply the patch and it should go through without issues.

Solution B: If you're already screwed up

If your instance is already patched and “now broken” you can probably file a SR and mention that you’re running into the bug 36040277; the dev team can then provide you a temporary package that allows to change the APPS Password and thus repair the instance / the Vault.

Summary

Beside of this, all the patches as part of 12.2.13, AD/TXK 15 and ECC V11 could be applied without any issues – which proves again that the entire process and system in the meantime is very stable and those yearly RUPs are really not too big a deal to apply.